Configuration Best Practices
This list distills how mature teams handle configuration and secrets for containerized workloads: one immutable image, config injected at run time, secrets never in Git, and a clear reload strategy.
How to Use This List
Read it top to bottom once to calibrate, then treat each group as a review checklist for a service.
Adopt the whole first group before anything else - it is foundational.
Do not chase every item on day one. Prioritize the security group for anything handling real credentials.
A - Separate Config From the Image
Build one image, promote it everywhere. Tag by immutable digest and inject per-environment values at run time so the same artifact runs in dev, staging, and prod.
Never bake secrets into images. ENV and ARG values persist in layers and image history. Use BuildKit secret mounts at build time and Kubernetes Secrets at run time.
Keep non-secret defaults in the image. Ship sane fallbacks like LOG_LEVEL=info so the image runs standalone, and let Pod-level config override them.
Apply the open-source test. If you could publish the image without leaking anything, your secrets are correctly externalized. If not, fix that first.
B - Structure Config in Kubernetes
Put non-secret config in ConfigMaps and sensitive values in Secrets. Keep the two object types distinct so RBAC and encryption apply only where needed.
Prefer explicit key references over bulk envFrom. configMapKeyRef and secretKeyRef make the container's contract reviewable and avoid surprise variables.
Mount files for config that must reload live. Volume mounts are updated in place by the kubelet, while env vars are frozen at container start.
Make stable config immutable. Set immutable: true on ConfigMaps and Secrets that rarely change to prevent accidental edits and reduce apiserver watch load.
C - Secure Secrets
Enable encryption at rest for Secrets. Base64 in etcd is not protection. A KMS provider is the single most important hardening step.
Scope Secret access with tight RBAC. Grant get on specific resourceNames only, and avoid broad list or watch on Secrets, which return the data itself.
Never commit raw Secret YAML. Private repos are not encrypted. Use Sealed Secrets or the External Secrets Operator so Git holds only ciphertext or references.
Mount high-value secrets as files with mode 0400. This keeps them out of the process environment, logs, and crash dumps that often capture env vars.
Rotate credentials on a schedule. Drive rotation from an external store and propagate with a reload strategy so no secret lives forever.
D - Version and Deliver Config With GitOps
Version ConfigMaps in Git and reconcile with Argo CD or Flux. This gives an audit trail, review, and one-command rollback for configuration.
Avoid live edits with kubectl edit. Manual edits cause drift from Git. Change the source of truth and let the reconciler apply it.
Reference external secret stores instead of storing values. With ESO or the CSI driver, the repo carries a pointer, and the real secret stays in Vault or a cloud manager.
Template environment differences, do not fork them. Use Helm values or Kustomize overlays so environments share one base and differ only by explicit patches.
E - Reload and Roll Out Safely
Choose a reload pattern that matches how the app reads config. File-watching or SIGHUP for daemons, and rolling restart for env-var consumers.
Use a checksum annotation or a reloader for env-var config. A config hash in the Pod template triggers a rolling update whenever values change.
Avoid subPath mounts when you need live updates. subPath files are frozen at Pod start and require a restart to change.
Guard rollouts with probes and a PodDisruptionBudget. Readiness probes and a PDB keep the service available while config-driven restarts roll through.
When You Are Done
Confirm one image promotes across all environments with only injected config differing.
Verify no secret exists in Git, image layers, or plaintext etcd, and that encryption at rest is on.
Check that every service has a defined reload strategy and that config lives in Git under GitOps reconciliation.
FAQs
What is the single most important practice here? Separate config from the image and never bake secrets in. Everything else builds on an immutable, secret-free artifact.
Do I need GitOps to follow these practices?
No, but versioning config in Git with Argo CD or Flux gives you audit, review, and rollback that ad-hoc kubectl apply cannot.
Env vars or file mounts by default? Use env vars for a few scalar values and file mounts for config files, certificates, and anything that must reload without a restart.
How do I keep secrets in Git safely? Encrypt them with Sealed Secrets, or keep the source of truth in an external store and reference it with the External Secrets Operator.
Is enabling encryption at rest really necessary? Yes. Without it, every etcd snapshot and backup is a plaintext credential archive that base64 does nothing to protect.
Related
- Separating Config and Secrets from Images
- Config Basics
- Secrets Management
- External Secrets Operator
- Config Reload Patterns
Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).