CI/CD Best Practices

These are the practices that keep Kubernetes delivery safe, reproducible, and auditable. They assume a build-scan-sign-commit-sync pipeline with a GitOps controller reconciling the cluster.

How to Use This List

Read the groups in order; they roughly follow a request from commit to running Pod. Adopt the git-as-source-of-truth group first, then layer on supply-chain, rollout, and operational practices.

Treat each bold item as a rule with a rationale. Where a rule is aspirational for your team, make it a tracked gap rather than a silent exception.

A - Git as the Source of Truth

No kubectl apply from laptops to production. Manual applies create untracked state that no one can review or reproduce. All changes flow through git and a controller.
Every deploy is a commit. The commit is your audit trail, your approval record, and your rollback point in one artifact.
Roll back with git revert. Reverting the deploy commit reconciles the previous state; avoid out-of-band hotfixes that git does not know about.
Enable drift detection and self-heal. Let Argo CD or Flux flag and revert manual changes so the cluster cannot silently diverge from git.
Separate app code from manifests. Keep deployment manifests in their own repo or path so a config change does not require an app rebuild.

B - Supply Chain and Image Hygiene

Deploy by digest, never by a moving tag. A @sha256: reference is immutable, so the running Pod is deterministic and rollback is exact.
Scan every image and gate on severity. Run Trivy or Grype in CI with a non-zero exit on HIGH and CRITICAL findings so vulnerable images never ship.
Sign images with cosign and verify at admission. Keyless Sigstore signing plus a Kyverno or Sigstore policy controller ensures only pipeline-produced images run.
Pin and minimize base images. Use small, pinned, non-root bases (distroless or slim) to shrink the attack surface and the CVE count.
Generate an SBOM. Produce a software bill of materials at build time so you can answer "are we affected" fast when the next CVE lands.

C - Pipeline Structure and Credentials

CI builds; the cluster deploys. Keep CI free of cluster API credentials; a pull-based controller inside the cluster does the applying.
Use short-lived, scoped credentials. Prefer OIDC federation for registry and cloud access over long-lived static secrets in CI.
Build with BuildKit and cache deliberately. The default BuildKit backend in Docker Engine 29.x speeds builds; scope caches so they never leak secrets between builds.
Fail fast and fail loud. Order stages so cheap checks (lint, unit tests) run before expensive ones (integration, scan), and make gate failures block the merge.
Keep the runtime boundary straight. Docker builds images in CI; containerd via the CRI runs Pods on nodes. Do not assume Docker Engine runs in-cluster.

D - Progressive and Safe Rollouts

Set requests, limits, and probes on every workload. Requests let the scheduler place Pods; limits cap blast radius; readiness probes gate traffic during rollouts.
Promote through environments, do not rebuild. Ship the exact digest that passed staging to production so the artifact never changes on the way up.
Gate risky services with metric-based canaries. Argo Rollouts or Flagger shift traffic gradually and auto-abort on bad success-rate or latency metrics.
Automate rollback on failed gates. A rollout that cannot roll itself back on bad metrics is just a slow deploy; wire the abort path.
Keep production behind a human promotion. Auto-update staging with an image bot, but require a reviewed PR to promote to production.

E - Security and Operations Defaults

Apply the restricted Pod Security Standard. Enforce runAsNonRoot, RuntimeDefault seccomp, and dropped capabilities at the namespace level.
Default-deny network traffic. Start with a deny-all NetworkPolicy and open only the flows each workload needs.
Instrument delivery with metrics and traces. Export rollout and error metrics to Prometheus and traces via OpenTelemetry so gates and dashboards have real signal.
Make secrets first-class and external. Use sealed secrets or an external secrets operator so credentials are never committed in plaintext to git.
Rehearse rollback and disaster recovery. A rollback path that has never been tested is a guess; exercise revert and restore regularly.

You should be able to trace any running Pod back to a commit, a scanned and signed image digest, and a reviewed deploy. No one should need laptop access to the production API server to ship or revert.

If any of that is not true yet, treat the gap as the next piece of work and track it openly.

FAQs

Why ban kubectl apply to production? It creates state that is not in git, so it is invisible to review, drift detection, and rollback. Route every change through git.

Tag or digest for deploys? Always digest. Tags move; a digest is content-addressed and makes rollouts and rollbacks deterministic.

Should CI have cluster credentials? No. In a GitOps flow CI only builds and commits, and an in-cluster controller pulls and applies, which removes a large credential blast radius.

Is scanning enough, or do I also sign? Both. Scanning checks content for CVEs; signing proves origin so admission control can reject anything your pipeline did not produce.

When is progressive delivery worth it? For high-traffic or high-blast-radius services where a plain rolling update cannot catch regressions before they reach many users.

How do I stop a bad auto-update? Revert the bot's commit in git; the GitOps controller reconciles the previous digest back into the cluster.

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).

CI/CD Best Practices

These are the practices that keep Kubernetes delivery safe, reproducible, and auditable. They assume a build-scan-sign-commit-sync pipeline with a GitOps controller reconciling the cluster.

How to Use This List

Read the groups in order; they roughly follow a request from commit to running Pod. Adopt the git-as-source-of-truth group first, then layer on supply-chain, rollout, and operational practices.

Treat each bold item as a rule with a rationale. Where a rule is aspirational for your team, make it a tracked gap rather than a silent exception.

A - Git as the Source of Truth

No kubectl apply from laptops to production. Manual applies create untracked state that no one can review or reproduce. All changes flow through git and a controller.
Every deploy is a commit. The commit is your audit trail, your approval record, and your rollback point in one artifact.
Roll back with git revert. Reverting the deploy commit reconciles the previous state; avoid out-of-band hotfixes that git does not know about.
Enable drift detection and self-heal. Let Argo CD or Flux flag and revert manual changes so the cluster cannot silently diverge from git.
Separate app code from manifests. Keep deployment manifests in their own repo or path so a config change does not require an app rebuild.

B - Supply Chain and Image Hygiene

Deploy by digest, never by a moving tag. A @sha256: reference is immutable, so the running Pod is deterministic and rollback is exact.
Scan every image and gate on severity. Run Trivy or Grype in CI with a non-zero exit on HIGH and CRITICAL findings so vulnerable images never ship.
Sign images with cosign and verify at admission. Keyless Sigstore signing plus a Kyverno or Sigstore policy controller ensures only pipeline-produced images run.
Pin and minimize base images. Use small, pinned, non-root bases (distroless or slim) to shrink the attack surface and the CVE count.
Generate an SBOM. Produce a software bill of materials at build time so you can answer "are we affected" fast when the next CVE lands.

C - Pipeline Structure and Credentials

CI builds; the cluster deploys. Keep CI free of cluster API credentials; a pull-based controller inside the cluster does the applying.
Use short-lived, scoped credentials. Prefer OIDC federation for registry and cloud access over long-lived static secrets in CI.
Build with BuildKit and cache deliberately. The default BuildKit backend in Docker Engine 29.x speeds builds; scope caches so they never leak secrets between builds.
Fail fast and fail loud. Order stages so cheap checks (lint, unit tests) run before expensive ones (integration, scan), and make gate failures block the merge.
Keep the runtime boundary straight. Docker builds images in CI; containerd via the CRI runs Pods on nodes. Do not assume Docker Engine runs in-cluster.

D - Progressive and Safe Rollouts

Set requests, limits, and probes on every workload. Requests let the scheduler place Pods; limits cap blast radius; readiness probes gate traffic during rollouts.
Promote through environments, do not rebuild. Ship the exact digest that passed staging to production so the artifact never changes on the way up.
Gate risky services with metric-based canaries. Argo Rollouts or Flagger shift traffic gradually and auto-abort on bad success-rate or latency metrics.
Automate rollback on failed gates. A rollout that cannot roll itself back on bad metrics is just a slow deploy; wire the abort path.
Keep production behind a human promotion. Auto-update staging with an image bot, but require a reviewed PR to promote to production.

E - Security and Operations Defaults

Apply the restricted Pod Security Standard. Enforce runAsNonRoot, RuntimeDefault seccomp, and dropped capabilities at the namespace level.
Default-deny network traffic. Start with a deny-all NetworkPolicy and open only the flows each workload needs.
Instrument delivery with metrics and traces. Export rollout and error metrics to Prometheus and traces via OpenTelemetry so gates and dashboards have real signal.
Make secrets first-class and external. Use sealed secrets or an external secrets operator so credentials are never committed in plaintext to git.
Rehearse rollback and disaster recovery. A rollback path that has never been tested is a guess; exercise revert and restore regularly.

When You Are Done

If any of that is not true yet, treat the gap as the next piece of work and track it openly.

FAQs

Why ban kubectl apply to production? It creates state that is not in git, so it is invisible to review, drift detection, and rollback. Route every change through git.

Tag or digest for deploys? Always digest. Tags move; a digest is content-addressed and makes rollouts and rollbacks deterministic.

Should CI have cluster credentials? No. In a GitOps flow CI only builds and commits, and an in-cluster controller pulls and applies, which removes a large credential blast radius.

Is scanning enough, or do I also sign? Both. Scanning checks content for CVEs; signing proves origin so admission control can reject anything your pipeline did not produce.

When is progressive delivery worth it? For high-traffic or high-blast-radius services where a plain rolling update cannot catch regressions before they reach many users.

How do I stop a bad auto-update? Revert the bot's commit in git; the GitOps controller reconciles the previous digest back into the cluster.

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).

CI/CD Best Practices

Search Documentation

CI/CD Best Practices