Why Governance Scales Where Heroics Don't
Summary
- Governance is the set of shared, enforced rules that make a container platform predictable across many teams, without a human reviewing every change.
- Insight: Heroics scale linearly with the number of experts; governance scales with the number of policies, which is a far smaller and slower-growing set.
- Key Concepts: paved road, policy as code, version policy, deprecation tracking, golden images, guardrails vs gates.
- When to Use: The moment you have more than a handful of teams shipping to Kubernetes and one person can no longer be in the loop on every Dockerfile and manifest.
- Limitations/Trade-offs: Governance adds friction and needs investment; over-rigid rules push teams to shadow platforms.
- Related Topics: org-wide Dockerfile and manifest standards, Kubernetes version policy, API deprecation tracking, add-on governance.
Foundations
Heroics is the model where a few deep experts personally keep the platform safe.
They review manifests, remember which APIs are being removed in Kubernetes 1.37, and hand-patch the one Deployment that forgot resource limits.
This works at three teams and collapses at thirty.
The expert becomes a bottleneck, then a single point of failure, then a burned-out person who leaves and takes the institutional memory with them.
Governance replaces the expert-in-the-loop with rules-in-the-pipeline.
A rule such as "every image must run as non-root" is written once, enforced automatically, and applies identically to team one and team fifty.
The mental model is a paved road: the easy path is also the compliant path.
Teams get scaffolding, base images, CI templates, and admission policies that make the right thing the default and the wrong thing hard.
Governance is not a review board that says no.
It is the accumulated set of defaults, templates, and automated checks that let hundreds of engineers move fast without each one needing to be a Kubernetes expert.
Mechanics & Interactions
Governance operates at three enforcement points, and they compound.
First, authoring time: golden base images, Dockerfile linters, and Helm/Kustomize templates shape what engineers write before they commit.
Second, CI time: image scanning (Trivy or Grype), manifest validation (kubeconform), and signing (cosign) run in the pipeline and fail the build on violations.
Third, admission time: the API server rejects non-compliant objects using a policy engine such as Kyverno or an OPA/Gatekeeper validating webhook, plus built-in Pod Security Admission.
A concrete example: a rule that pods must set CPU and memory requests.
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: require-requests
spec:
validationFailureAction: Enforce
rules:
- name: check-requests
match:
any:
- resources:
kinds: ["Pod"]
validate:
message: "CPU and memory requests are required."
pattern:
spec:
containers:
- resources:
requests:
cpu: "?*"
memory: "?*"The same policy protects every namespace, forever, with no reviewer required.
The distinction that matters is guardrails vs gates.
Guardrails are non-blocking signals: a scan warning, a lint comment, a dashboard.
Gates are blocking: admission denial, a failed CI step, a rejected pull request.
Effective governance starts most rules as guardrails (Audit mode), watches the violation rate, then promotes them to gates (Enforce) once teams have migrated.
This is how you avoid the failure mode where a well-meaning policy takes down production on the day it ships.
Advanced Considerations & Applications
The hardest part of governance is not writing policies; it is versioning and deprecating them.
Kubernetes itself models this well with its version policy: nodes run within N-1 of the control plane, minors are supported for roughly a year, and APIs move through alpha, beta, and stable with removal announced in advance.
Your platform rules deserve the same discipline: publish them, version them, and give teams a migration window before a guardrail becomes a gate.
Deprecation tracking is the counterpart to version policy.
When Kubernetes removes an API version - as it did with PodSecurityPolicy and older Ingress versions - governance is what turns a surprise outage into a scheduled migration.
Tools like kubent (kube-no-trouble) and Pluto scan live clusters and manifests for API versions that will be removed next upgrade.
At scale, governance also owns the add-on catalog: the approved list of Helm charts and operators, each pinned, scanned, and assigned an owner.
Without it, every team installs its own ingress controller and cert-manager, and you inherit fifteen slightly different, unpatched copies.
The economic argument is simple.
Heroics has O(teams) cost because each new team needs expert attention; governance has O(policies) cost because a new team inherits every existing rule for free.
That crossover is why platform organizations invest in policy engines, golden images, and GitOps rather than hiring more reviewers.
Common Misconceptions
"Governance means a central team approves every deploy."
No. Good governance removes humans from the routine path; approval is reserved for genuine exceptions.
"Policies slow teams down."
Poorly rolled-out policies do. Guardrail-first rollout with a migration window makes the compliant path the fastest path.
"We can enforce standards with documentation and code review."
Documentation is not enforcement. If a rule is not checked by CI or admission, it will drift within weeks.
"Governance is the same as security."
Security is one input. Governance also covers reliability, cost, version currency, and consistency, which are not security concerns.
"Once written, policies are done."
Policies rot. They need version policy, deprecation windows, and periodic review just like the software they govern.
FAQs
Where should I enforce a rule: CI or admission?
Both. CI gives fast, early feedback to the author; admission is the backstop that catches anything applied out-of-band, including via GitOps or kubectl.
Kyverno or OPA/Gatekeeper?
Kyverno uses Kubernetes-native YAML and is easier for most teams; Gatekeeper uses Rego and suits complex logic. Pick one and standardize.
How do I introduce a policy without breaking production?
Ship it in Audit or Warn mode, measure violations, help teams migrate, then flip to Enforce with a published cutover date.
Does governance replace platform engineers?
No. It changes their job from reviewing every change to building and maintaining the paved road and its policies.
What is the smallest useful first policy?
Require resource requests and limits, and enforce Pod Security Standards at the restricted or baseline level.
How does this relate to Kubernetes version support?
Version policy is governance applied to the cluster itself: it sets when you upgrade and how far nodes may lag the control plane.
Related
- Governance Basics
- Kubernetes Version Policy
- API Deprecation Tracking
- Extension & Add-On Governance
- Governance Best Practices
Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).