Secrets Management

Summary

• Definition: Secrets management is the discipline of storing, injecting, and rotating sensitive values so they never live in Git, image layers, or plaintext logs.
• Insight: A Kubernetes Secret is only base64-encoded in etcd by default. Encryption at rest plus RBAC is what makes it a real secret.
• Key Concepts: encryption at rest, RBAC scoping, Sealed Secrets, External Secrets Operator, rotation.
• When to Use: For every credential, token, TLS key, or connection string a workload needs.
• Limitations: Native Secrets have no rotation, no versioning, and no audit trail on their own. Those come from external tooling.

Recipe

The minimal safe path for a small team.

1. Enable encryption at rest for secrets on the API server so etcd never holds plaintext.
2. Create Secrets imperatively, not from committed YAML.
3. Scope read access with a Role and RoleBinding so only the workload's ServiceAccount can read the Secret.
4. Consume the Secret as a mounted file where sensitivity is high, or a secretKeyRef env var otherwise.
5. To version Secrets in Git, encrypt them first with Sealed Secrets, or reference an external store with ESO.

Working Example

Create a Secret without leaving plaintext in your shell history or Git.

# Create the Secret directly - no YAML file to accidentally commit
kubectl create secret generic app-db \
  --from-literal=password="$(openssl rand -base64 24)" \
  --namespace app

Restrict who can read it with RBAC.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: read-app-db
  namespace: app
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["app-db"]
    verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: app-reads-db
  namespace: app
subjects:
  - kind: ServiceAccount
    name: app
    namespace: app
roleRef:
  kind: Role
  name: read-app-db
  apiGroup: rbac.authorization.k8s.io

Mount it as a file so the value stays out of the environment and process listing.

spec:
  serviceAccountName: app
  containers:
    - name: api
      image: registry.example.com/app@sha256:abc123
      volumeMounts:
        - name: db
          mountPath: /etc/secrets/db
          readOnly: true
  volumes:
    - name: db
      secret:
        secretName: app-db
        defaultMode: 0400

To keep a versioned, Git-native secret, seal it so only the cluster controller can decrypt.

# Sealed Secrets: encrypt with the controller's public cert, safe to commit
kubeseal --format yaml < app-db-secret.yaml > app-db-sealed.yaml

Deep Dive

How Secrets are stored

By default the API server writes Secret data to etcd base64-encoded. Anyone with etcd or backup access can decode it trivially.

Turning on EncryptionConfiguration with a provider such as KMS encrypts Secret data before it reaches etcd. This is the single most important hardening step.

RBAC is the access boundary

A Secret is readable by any principal with get on Secrets in its namespace. Scope Roles to specific resourceNames so a compromised workload cannot read every credential.

Avoid list and watch on Secrets in broad Roles. Those verbs return the data, not just names.

File mounts versus env vars

Mounted Secret files can be set to mode 0400 and stay out of the process environment, crash dumps, and child processes.

Env-var Secrets are convenient but leak more easily into logs and /proc. Prefer files for high-value credentials.

Sealed Secrets versus External Secrets

Sealed Secrets encrypt a Secret with the cluster controller's public key, so the ciphertext is safe in Git and only that cluster can decrypt it.

The External Secrets Operator instead keeps the source of truth in Vault or a cloud manager and syncs it into a Kubernetes Secret. That path adds rotation and central audit.

Gotchas

Committing raw Secret YAML to Git is the classic mistake. Private repos are not encrypted, and a single clone or leaked token exposes everything.

Base64 tricks people into thinking Secrets are encrypted. kubectl get secret app-db -o jsonpath='{.data.password}' | base64 -d reveals the value instantly.

Env-var Secrets appear in application logs when frameworks dump the environment on error. Mount sensitive values as files to avoid this.

Forgetting encryption at rest means every etcd backup is a plaintext credential archive. Enable it before you store anything real.

Overly broad RBAC undermines everything else. A Role granting get on all Secrets in a namespace defeats per-Secret isolation.

Alternatives

Native Secrets with encryption at rest and tight RBAC suit small clusters with few credentials and no rotation requirement.

Sealed Secrets fit GitOps teams that want Secrets versioned in Git without an external store to operate.

The External Secrets Operator fits organizations already running Vault or a cloud secret manager and needing rotation, versioning, and central audit.

CSI Secrets Store drivers mount secrets directly from an external store into the Pod without creating a Kubernetes Secret at all, which minimizes etcd exposure.

FAQs

Are Kubernetes Secrets encrypted? Not by default. They are base64-encoded in etcd until you enable encryption at rest with a KMS or aescbc provider.

Can I store Secret YAML in Git if the repo is private? No. Private is not encrypted. Use Sealed Secrets or the External Secrets Operator so Git holds only ciphertext or references.

Env var or file mount for a database password? Prefer a file mount with mode 0400. It keeps the value out of the process environment and out of most log paths.

How do I rotate a native Secret? Update the Secret, then trigger a rolling restart or use a reloader so consumers pick up the new value. Native Secrets have no automatic rotation.

What is the smallest hardening step that matters most? Enable encryption at rest for Secrets. Without it, every etcd backup is a plaintext credential dump.

Related

• Separating Config and Secrets from Images
• External Secrets Operator
• Config Reload Patterns
• Config Basics
• Configuration Best Practices

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).