GitOps Best Practices
These are the practices that keep a GitOps setup trustworthy at scale: a clear repository contract, safe secrets, correct ordering, and disciplined promotion.
They apply whether you run Argo CD or Flux on Kubernetes 1.36.2, with images built by Docker/BuildKit and run by containerd on the nodes.
How to Use This List
Read the groups top to bottom; each builds on the last.
Adopt them incrementally. Repository structure and secrets first, then reconciliation policy, then promotion and observability.
Treat every item as something a reviewer can check in a pull request, not a one-time setup step.
A - Repository Structure
Separate platform from application manifests. Keep cluster-wide concerns (ingress, CRDs, policy, operators) in a different repo or clearly separated folder from app manifests, so platform and app teams own distinct blast radii.
Adopt one explicit folder contract and document it. Whether it is repo-per-team, repo-per-environment, or a monorepo with clusters/<env> paths, write the contract down so every new service lands in a predictable place.
Use overlays, not copies, for environments. Kustomize base plus overlays/staging and overlays/prod keeps environments in sync while allowing per-environment patches for replicas or resource limits.
Keep the GitOps tool self-managed in Git. Bootstrap Flux or manage Argo CD's own config from the repo, so upgrades to the delivery system are themselves reviewed commits.
Pin images by digest or immutable tag. Reference image@sha256:... or an immutable tag so the deployed artifact is reproducible and cannot silently change under a floating tag.
B - Secrets and Supply Chain
Never commit plaintext secrets. Use Sealed Secrets, SOPS-encrypted files, or an external secrets operator so Git holds only ciphertext or references.
Scan images before they reach a manifest. Run Trivy or Grype in CI and fail the build on critical CVEs, so Git never pins a known-vulnerable image.
Sign images and verify at admission. Sign with cosign and enforce verification with a policy engine, so only trusted images run even if a manifest is tampered with.
Give the agent least-privilege Git and cluster access. Read-only deploy keys where write-back is not needed, and scoped RBAC via Argo CD AppProject or per-namespace Flux installs.
C - Reconciliation Policy
Enable pruning deliberately. Turn on prune so resources deleted from Git are removed from the cluster, but diff first so you do not delete something you forgot was unmanaged.
Use self-heal to enforce Git as truth. With selfHeal: true, manual kubectl edit changes are reverted automatically, which is the core promise covered in Drift Detection.
Do not declare fields other controllers own. Omit spec.replicas when an HPA manages it, or the agent and the autoscaler will overwrite each other every loop.
Order resources explicitly. Use Argo CD sync waves or Flux dependsOn so CRDs and namespaces apply before the resources that need them.
Set health checks and timeouts. Flux wait: true with healthChecks, or Argo CD health status, so a sync reports success only when workloads are actually ready.
D - Promotion and Change Flow
Make every change a pull request. All desired-state changes flow through review, giving you an audit trail and a natural approval gate.
Gate production sync. Auto-sync lower environments, but require manual sync or an approval for production so a bad commit does not roll out unreviewed.
Promote by moving a known-good artifact forward. Copy the exact image digest from staging to the prod overlay rather than rebuilding, so you deploy what you already tested.
Roll back with git revert. Recover by reverting the commit and letting the agent reconcile, rather than hand-editing the cluster during an incident.
Automate image bumps with a write-back to Git. Use Flux image automation or Argo CD Image Updater so new tags land as reviewable commits, not out-of-band changes.
E - Observability and Operations
Alert on OutOfSync and failed reconciliations. Wire notifications to Slack or your incident tool so drift and sync failures are visible immediately.
Export and watch reconciliation metrics. Scrape Argo CD and Flux Prometheus metrics for sync duration, failure counts, and reconciliation lag.
Establish a break-glass procedure. Define how to make an emergency change and immediately codify it back to Git, so self-heal does not erase a live fix.
Test manifests in CI before merge. Run kubectl diff, kustomize build, and policy checks in the pipeline so broken YAML never reaches the agent.
When You Are Done
You should have a documented repo contract, encrypted secrets, explicit reconciliation policy, and a reviewed promotion path.
Verify it by making a change only through Git and confirming the cluster converges, then by hand-editing a resource and watching self-heal revert it.
If a manual cluster change survives longer than one reconciliation interval, your setup is not yet enforcing Git as the source of truth.
FAQs
One repo or many? Either works; what matters is a documented contract. Many teams split platform and app repos for ownership and blast-radius reasons.
Should production auto-sync? Usually not. Gate production behind manual sync or approval while auto-syncing lower environments for speed.
How do I promote between environments? Move the tested image digest into the target overlay via a pull request, rather than rebuilding for each environment.
Where do secrets go? Encrypted in Git via Sealed Secrets or SOPS, or referenced through an external secrets operator, never in plaintext.
Argo CD or Flux for these practices? Both support all of them. See Argo CD and Flux; the practices are tool-agnostic.
What runs the images in production? containerd via CRI on the nodes; Docker/BuildKit only builds them. GitOps decides which image runs where.
Related
Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).