Governance Best Practices

Manual audits work at three teams, strain at ten, and collapse at thirty. Everything below is about converting judgment into automation that runs without you.

How to Use This List

Read it once end to end, then pick the two practices whose absence is currently costing you the most.

Do not adopt all of them at once. Governance that arrives as a wall of new blocks gets routed around, and a bypassed control is worse than no control because it lies to you.

Every practice here should end up as code in a repo: a policy, a gate, a chart, or a scheduled check. If it ends up as a wiki page, it did not happen.

Put the standard in a policy engine, not a review checklist. A Kyverno or Gatekeeper rule applies to the manifest nobody reviewed at 2am. A checklist applies to the ones with an attentive reviewer.

Ship defaults as a Helm library chart. Teams should get runAsNonRoot, seccomp, and probes by bumping a dependency version, not by copying YAML between repos.

Version your standards and publish a changelog. A standard is an API for your org. Breaking it silently is as rude as breaking a library API silently.

Make the compliant path the shortest path. If the secure template requires forty extra lines, teams will not use it. Governance loses every fight with convenience.

Validate schemas in CI with the production version pinned. kubeconform -strict -kubernetes-version 1.36.2 catches typos and unknown fields offline, in seconds, before a cluster is involved.

B - Enforce at Admission, Not in Review

Start every policy in Audit mode. Read the policy reports for at least a week. The gap between what you think is deployed and what is deployed is always larger than expected.

Roll out enforcement by namespace, not by fleet. Land it on one team, learn what breaks, then widen. A cluster-wide Enforce on day one is a self-inflicted outage.

Use Pod Security admission for the baseline and a policy engine for the rest. PSA is built in, free, and handles the restricted profile. Do not rebuild it in Kyverno.

Pin pod-security.kubernetes.io/enforce-version. Without the pin, a cluster upgrade can silently tighten what the restricted profile means and reject workloads that used to pass.

Scope webhooks with a namespaceSelector that excludes kube-system and your platform namespaces. A webhook with failurePolicy: Fail and no healthy backend can block the deploy that would fix it.

Keep webhook timeoutSeconds low. Every admission webhook sits in the API server's request path. A slow webhook is everyone's latency problem.

C - Govern Versions on a Calendar

Write down a target minor and a floor minor. Today that is target 1.36.2, floor 1.34. The supported window is three minors and 1.33 is EOL with no patches coming.

Upgrade the control plane before the nodes, always. A kubelet newer than the API server is unsupported. Keep kubelets within one minor as policy even though upstream allows more.

Never skip a minor. One at a time, or you skip the deprecation warnings that would have told you what breaks.

Make the calendar produce more upgrades per year than upstream produces releases. Roughly three minors ship annually. Any slower cadence drifts you into EOL by arithmetic.

Separate patch upgrades from minor upgrades in policy. 1.36.1 to 1.36.2 should be routine. Do not let minor-upgrade planning hold a CVE fix hostage.

Treat vendor extended support as recovery, not strategy. It buys calendar time past upstream EOL. It does not buy safety, and it costs real money.

D - Track Deprecations Continuously

Alert on apiserver_requested_deprecated_apis > 0. The API server already tells you which group, version, and resource is in use, and which release removes it. Almost nobody reads it.

Gate upgrades on removed_release. If any deprecated API in use has a removed_release at or below the minor you are installing, the upgrade stops. The exception is an outage.

Scan Git, not just the live cluster. The metric only fires when something calls. A quarterly CronJob on a removed API passes every live scan and fails the upgrade.

Render vendor charts before you grep them. The deprecated apiVersion lives in the chart's templates, not your repo. helm template first.

Check webhooks and RBAC for removed API groups too. A webhook that stops matching does not error - it just quietly stops enforcing. That is a policy failing open.

Migrate CRD stored versions before dropping old versions. Removing a version from spec.versions while objects are still stored at it makes them unreadable, with no undo.

E - Govern Add-Ons Like Infrastructure

Publish an approved catalog in Git with an owner per entry. An operator holds a ClusterRole and watches every namespace. An unowned add-on is an unowned outage.

Record a supportedKubernetes range for every add-on. Your version policy and your catalog are one system. A critical add-on without 1.36 support blocks the fleet's 1.36 upgrade.

Mirror charts and images into an internal registry and pin by digest. Upstream registries yank charts and rename orgs. If you did not mirror it, your rebuild depends on someone else's retention policy.

Track CRD API groups in the catalog and refuse collisions. Two operators claiming monitoring.coreos.com cannot coexist, and the second install can overwrite what the first depends on.

Read the ClusterRole before you install. An add-on requesting * on * in * is a cluster-admin backdoor shipped as a Helm chart.

Tier add-ons by blast radius and give lower tiers lighter process. CNI and ingress deserve gates and PDBs. A team-scoped dashboard does not, and over-processing it teaches people to bypass you.

F - Govern the Image Supply Chain

Pin base images by digest, not tag. A floating tag makes every rebuild a lottery. Digests make builds reproducible and scans meaningful.

Scan with --exit-code 1 or do not scan. A Trivy report that never fails a build is decoration. Gate on HIGH and CRITICAL, and triage the rest on a schedule.

Scan and sign the same digest. Scan by digest, sign that digest with cosign, and verify signatures at admission so an unsigned image cannot be scheduled.

Keep the Docker and containerd boundary straight in your standards. Docker Engine 29.6.1 with BuildKit builds images in CI. containerd runs pods on nodes via CRI. Dockershim is gone and Docker Engine is not your in-cluster runtime.

Require owner metadata on every workload. Use the app.kubernetes.io/* common labels plus an owner annotation. Ownership is what turns a scanner report into a ticket someone answers.

When You Are Done

You should be able to answer four questions without opening a spreadsheet.

Which minors is every cluster on, and how far from the floor? Which deprecated APIs are in use, and do any of them get removed by our next upgrade?

Which add-ons are installed that are not in the catalog, and who owns each one? Which workloads have no owner annotation, no memory limit, or no probes?

If a script answers all four, your governance scales. If a person answers them, you have a bus factor.

The last check is the honest one. Turn one enforcement off for a day and see whether anything regresses. If nothing does, the control was theater. If everything does, you found the control that was carrying the whole system alone.

FAQs

Where do we start if we have nothing? Pod Security admission labels in warn mode on every namespace, and an alert on apiserver_requested_deprecated_apis. Both are free, built in, and take an afternoon.

Audit or Enforce for a new policy? Audit for at least a week, always. The reports will surprise you, and surprises are cheaper before the block is live.

How do we stop teams from routing around governance? Make compliance the default in the library chart so the easy path is the correct path. Enforcement should only catch the people who went out of their way.

Does governance slow teams down? Manual governance does. Automated governance is faster than the alternative, which is finding out during an upgrade window.

Who owns all of this? The platform team owns the calendar, the gates, and the catalog. Application teams own fixing their own deprecated APIs and resource settings before the window opens.

How often should the catalog be reviewed? On the same calendar as your Kubernetes upgrades. Sunset anything with no owner or no upstream release in twelve months.

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).

Governance Best Practices

Manual audits work at three teams, strain at ten, and collapse at thirty. Everything below is about converting judgment into automation that runs without you.

How to Use This List

Read it once end to end, then pick the two practices whose absence is currently costing you the most.

Do not adopt all of them at once. Governance that arrives as a wall of new blocks gets routed around, and a bypassed control is worse than no control because it lies to you.

Every practice here should end up as code in a repo: a policy, a gate, a chart, or a scheduled check. If it ends up as a wiki page, it did not happen.

A - Encode Standards as Code

Ship defaults as a Helm library chart. Teams should get runAsNonRoot, seccomp, and probes by bumping a dependency version, not by copying YAML between repos.

Version your standards and publish a changelog. A standard is an API for your org. Breaking it silently is as rude as breaking a library API silently.

Make the compliant path the shortest path. If the secure template requires forty extra lines, teams will not use it. Governance loses every fight with convenience.

Validate schemas in CI with the production version pinned. kubeconform -strict -kubernetes-version 1.36.2 catches typos and unknown fields offline, in seconds, before a cluster is involved.

B - Enforce at Admission, Not in Review

Start every policy in Audit mode. Read the policy reports for at least a week. The gap between what you think is deployed and what is deployed is always larger than expected.

Roll out enforcement by namespace, not by fleet. Land it on one team, learn what breaks, then widen. A cluster-wide Enforce on day one is a self-inflicted outage.

Use Pod Security admission for the baseline and a policy engine for the rest. PSA is built in, free, and handles the restricted profile. Do not rebuild it in Kyverno.

Pin pod-security.kubernetes.io/enforce-version. Without the pin, a cluster upgrade can silently tighten what the restricted profile means and reject workloads that used to pass.

Keep webhook timeoutSeconds low. Every admission webhook sits in the API server's request path. A slow webhook is everyone's latency problem.

C - Govern Versions on a Calendar

Write down a target minor and a floor minor. Today that is target 1.36.2, floor 1.34. The supported window is three minors and 1.33 is EOL with no patches coming.

Upgrade the control plane before the nodes, always. A kubelet newer than the API server is unsupported. Keep kubelets within one minor as policy even though upstream allows more.

Never skip a minor. One at a time, or you skip the deprecation warnings that would have told you what breaks.

Make the calendar produce more upgrades per year than upstream produces releases. Roughly three minors ship annually. Any slower cadence drifts you into EOL by arithmetic.

Separate patch upgrades from minor upgrades in policy. 1.36.1 to 1.36.2 should be routine. Do not let minor-upgrade planning hold a CVE fix hostage.

Treat vendor extended support as recovery, not strategy. It buys calendar time past upstream EOL. It does not buy safety, and it costs real money.

D - Track Deprecations Continuously

Alert on apiserver_requested_deprecated_apis > 0. The API server already tells you which group, version, and resource is in use, and which release removes it. Almost nobody reads it.

Gate upgrades on removed_release. If any deprecated API in use has a removed_release at or below the minor you are installing, the upgrade stops. The exception is an outage.

Scan Git, not just the live cluster. The metric only fires when something calls. A quarterly CronJob on a removed API passes every live scan and fails the upgrade.

Render vendor charts before you grep them. The deprecated apiVersion lives in the chart's templates, not your repo. helm template first.

Check webhooks and RBAC for removed API groups too. A webhook that stops matching does not error - it just quietly stops enforcing. That is a policy failing open.

Migrate CRD stored versions before dropping old versions. Removing a version from spec.versions while objects are still stored at it makes them unreadable, with no undo.

E - Govern Add-Ons Like Infrastructure

Publish an approved catalog in Git with an owner per entry. An operator holds a ClusterRole and watches every namespace. An unowned add-on is an unowned outage.

Record a supportedKubernetes range for every add-on. Your version policy and your catalog are one system. A critical add-on without 1.36 support blocks the fleet's 1.36 upgrade.

Track CRD API groups in the catalog and refuse collisions. Two operators claiming monitoring.coreos.com cannot coexist, and the second install can overwrite what the first depends on.

Read the ClusterRole before you install. An add-on requesting * on * in * is a cluster-admin backdoor shipped as a Helm chart.

Tier add-ons by blast radius and give lower tiers lighter process. CNI and ingress deserve gates and PDBs. A team-scoped dashboard does not, and over-processing it teaches people to bypass you.

F - Govern the Image Supply Chain

Pin base images by digest, not tag. A floating tag makes every rebuild a lottery. Digests make builds reproducible and scans meaningful.

Scan with --exit-code 1 or do not scan. A Trivy report that never fails a build is decoration. Gate on HIGH and CRITICAL, and triage the rest on a schedule.

Scan and sign the same digest. Scan by digest, sign that digest with cosign, and verify signatures at admission so an unsigned image cannot be scheduled.

Require owner metadata on every workload. Use the app.kubernetes.io/* common labels plus an owner annotation. Ownership is what turns a scanner report into a ticket someone answers.

When You Are Done

You should be able to answer four questions without opening a spreadsheet.

Which minors is every cluster on, and how far from the floor? Which deprecated APIs are in use, and do any of them get removed by our next upgrade?

Which add-ons are installed that are not in the catalog, and who owns each one? Which workloads have no owner annotation, no memory limit, or no probes?

If a script answers all four, your governance scales. If a person answers them, you have a bus factor.

FAQs

Audit or Enforce for a new policy? Audit for at least a week, always. The reports will surprise you, and surprises are cheaper before the block is live.

Does governance slow teams down? Manual governance does. Automated governance is faster than the alternative, which is finding out during an upgrade window.

Who owns all of this? The platform team owns the calendar, the gates, and the catalog. Application teams own fixing their own deprecated APIs and resource settings before the window opens.

How often should the catalog be reviewed? On the same calendar as your Kubernetes upgrades. Sunset anything with no owner or no upstream release in twelve months.

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).

Governance Best Practices

Search Documentation

Governance Best Practices