Build Pipeline Best Practices

This list distills a production-grade image build pipeline into concrete, enforceable practices. It targets tech leads standardizing CI across services on Docker Engine 29.6.1 and Kubernetes 1.36.2.

How to Use This List

Treat each practice as a gate you can enforce, not advice you can ignore. Wire the ones marked as failures into CI so drift cannot creep back in.

Adopt them incrementally. Start with reproducibility and non-root images, then layer in scanning, caching, and provenance.

Review the list per service during onboarding. A new repo should inherit these defaults from a shared workflow, not reinvent them.

A - Reproducibility & Correctness

• Pin base images by digest. Use FROM node:22-slim@sha256:... so a floating tag cannot silently change your build. Renovate or Dependabot can bump the digest deliberately.
• Pin dependency versions. Commit lockfiles and use npm ci, pip install -r with hashes, or Go modules so installs are deterministic.
• Deploy by digest, not tag. The pipeline output is image@sha256:...; tags are human aliases. Digests guarantee exact bytes and clean rollback.
• Lint the Dockerfile and fail on classified errors. Run hadolint and fail CI on warnings you have decided are errors for your org, not just on its defaults.
• Keep one source of truth for the Dockerfile. Avoid per-environment Dockerfiles; parameterize with build args instead so every environment ships identical layers.

B - Image Hygiene & Size

• Use multi-stage builds. Compile in a build stage and copy only artifacts into a minimal runtime, leaving compilers and package managers behind.
• Run as non-root. Add USER and a numeric UID so the image satisfies restricted Pod Security Standards without a securityContext override.
• Choose minimal bases. Prefer slim, alpine, or distroless runtimes to shrink attack surface and speed node pulls.
• Add a .dockerignore. Exclude .git, node_modules, and secrets so build context stays small and nothing sensitive leaks into a layer.
• Set OCI labels. Use docker/metadata-action to stamp source, revision, and version so images are self-describing in the registry.

C - CI Flow & Gating

• Build on every PR, push only from main and tags. Validate the Dockerfile on pull requests with push: false; publish only from protected refs.
• Fail fast, cheap checks first. Order lint and unit tests before scan and integration so obvious failures return in seconds.
• Never expose credentials to fork PRs. Gate registry login and push on github.event_name == 'push' so untrusted forks cannot publish or leak secrets.
• Use OIDC for registry auth. Prefer short-lived tokens (GitHub GITHUB_TOKEN, cloud OIDC) over long-lived stored registry passwords.
• Make the pipeline the only publisher. Block human docker push to production repos so every image is traceable to a commit and workflow.

D - Caching & Speed

• Order instructions least-to-most volatile. Base, then dependency install, then source copy, so edits invalidate as few cached layers as possible.
• Use registry-backed remote cache with mode=max. Ephemeral runners then restore intermediate layers instead of rebuilding from cold.
• Scope cache per service in monorepos. Give each buildable target its own cache ref so unrelated builds do not evict each other.
• Build multi-arch on native runners when speed matters. Use QEMU only for light images; a native amd64/arm64 matrix plus a manifest merge is faster for compilation.
• Prune cache on a schedule. Registry cache grows unbounded; apply retention or GC so storage cost stays flat.

E - Security & Supply Chain

• Scan images and fail on HIGH/CRITICAL. Run Trivy or Grype after build and before push with a non-zero exit on actionable severities.
• Generate an SBOM per image. Attach it as an attestation so you can triage new CVEs against already-published images.
• Produce provenance and sign the digest. Enable BuildKit provenance and cosign keyless signing so origin is verifiable.
• Enforce signatures at admission. A Kyverno or Sigstore policy that rejects unsigned images turns attestations into a real gate, not a document.
• Re-scan published images periodically. CVEs are disclosed after build; a scheduled re-scan catches images that were clean at build time.

When You Are Done

You should be able to trace any running pod's digest back to a signed image, a commit, and a workflow run. If you cannot, tighten gating until you can.

Every practice above should be a check that fails CI or blocks admission - not a wiki page. Enforcement is what keeps standards alive across a growing fleet of services.

Re-audit quarterly. Base image digests, scanner databases, and SLSA tooling all move, so the pipeline needs periodic maintenance.

FAQs

Which practice should I adopt first? Pin bases by digest and deploy by digest. Reproducibility underpins every other guarantee, including rollback and provenance.

Is failing on all hadolint warnings too strict? Classify them. Promote the rules that matter to your org to errors and demote the noisy ones, then enforce that curated set.

Do small services need signing and provenance? Adopt them before the image crosses a trust boundary. Internal throwaway images can wait; anything a consumer verifies should be signed.

How do I stop humans pushing to prod repos? Restrict registry write permission to the CI identity and remove human write access, so the pipeline is the only publisher.

Should scanning block the build or just warn? Block on HIGH and CRITICAL that are actionable; warn on lower severities so the gate stays credible and unignored.

How often should I re-scan published images? On a schedule - daily or weekly - because new CVEs land after build. Pair it with SBOMs to target which images are affected.

Related

• What an Image Build Pipeline Is For
• Image CI Basics
• GitHub Actions & buildx
• Remote Cache
• Provenance & SLSA

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).