Golden Path Governance

Summary

• A golden path is the supported, opinionated way to ship a service on your platform - a versioned template plus the guardrails, defaults, and docs that come with it.
• Insight: A golden path only works if it is versioned software, not a wiki page. The moment teams cannot tell which version they are on, the path stops being golden and becomes folklore.
• Key Concepts: template versioning (SemVer on the chart or Kustomize base), deprecation windows (announce, warn, enforce), migration guides (mechanical steps, not prose), drift detection (who is off-path and by how much), escape hatches (the documented way to leave the path).
• When to Use: Once you have more than roughly five app teams sharing a cluster and you are answering the same "how do I add a Deployment?" question every sprint.
• Limitations: Golden paths cost real maintenance. If you cannot fund a template owner, you will ship a v1 that rots and teams will fork it anyway.

Recipe

Ship the path as a versioned Helm chart your teams depend on, not copy from.

1. Publish a library or wrapper chart (platform-service) to an OCI registry with SemVer tags.
2. Encode defaults in the chart: non-root securityContext, probes, requests/limits, PodDisruptionBudget, standard labels.
3. Require the chart version in each app's Chart.yaml dependency - that pin is your inventory.
4. Enforce the non-negotiables in admission (Kyverno/Gatekeeper) so off-path manifests fail regardless of template.
5. Deprecate on a published clock: announce at N-2, warn in CI at N-1, fail admission at N.

Working Example

A wrapper chart that gives teams a safe Deployment from ~10 lines of values.

# charts/platform-service/Chart.yaml
apiVersion: v2
name: platform-service
version: 3.4.0            # SemVer: teams pin this
appVersion: "1.0"

# charts/platform-service/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Release.Name }}
  labels:
    app.kubernetes.io/name: {{ .Release.Name }}
    app.kubernetes.io/managed-by: {{ .Release.Service }}
    platform.example.com/template-version: {{ .Chart.Version | quote }}
spec:
  replicas: {{ .Values.replicas | default 2 }}
  selector:
    matchLabels:
      app.kubernetes.io/name: {{ .Release.Name }}
  template:
    metadata:
      labels:
        app.kubernetes.io/name: {{ .Release.Name }}
        platform.example.com/template-version: {{ .Chart.Version | quote }}
    spec:
      securityContext:
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: app
          image: {{ required "image.repository is required" .Values.image.repository }}@{{ required "image.digest is required" .Values.image.digest }}
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
          resources:
            requests:
              cpu: {{ .Values.resources.requests.cpu | default "100m" }}
              memory: {{ .Values.resources.requests.memory | default "128Mi" }}
            limits:
              memory: {{ .Values.resources.limits.memory | default "512Mi" }}
          readinessProbe:
            httpGet: { path: /healthz/ready, port: 8080 }
            periodSeconds: 5
          livenessProbe:
            httpGet: { path: /healthz/live, port: 8080 }
            periodSeconds: 10
            failureThreshold: 6

The app team's entire input is small:

# apps/checkout/values.yaml
image:
  repository: registry.example.com/checkout
  digest: sha256:7d3f...c9
replicas: 4
resources:
  requests: { cpu: "500m", memory: "512Mi" }

Now find who is off-path. The template-version label makes drift a query:

kubectl get deploy -A \
  -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,TPL:.metadata.labels.platform\.example\.com/template-version' \
  | sort -k3

Anything with <none> never adopted the path. Anything on 2.x is your migration backlog.

Deep Dive

Version the template like an API

Treat the chart's values schema as a public API and apply SemVer honestly.

A patch fixes a bug in rendering. A minor adds an optional value with a safe default. A major removes or renames a value, or changes a default in a way that alters running workloads.

Changing a default is a breaking change. Flipping replicas from 1 to 2 silently doubles a team's cost, and they will find out from finance.

Add a values.schema.json so misuse fails at helm template time instead of at 2 a.m.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "required": ["image"],
  "properties": {
    "image": {
      "type": "object",
      "required": ["repository", "digest"],
      "properties": {
        "repository": { "type": "string" },
        "digest": { "type": "string", "pattern": "^sha256:[a-f0-9]{64}$" }
      }
    }
  }
}

Deprecate on a clock, not on a mood

Publish the window before you need it, so no deprecation feels personal.

Announce (N-2): the new major ships, release notes carry the migration guide, old version still fully supported.

Warn (N-1): the old chart renders a NOTES.txt deprecation banner and your CI check prints a warning with the removal date. Nothing breaks.

Enforce (N): admission rejects new deployments on the dead version. Existing workloads keep running - you are blocking new debt, not causing an outage.

Six to twelve weeks per stage is realistic for most orgs. Shorter than four weeks and you will be doing the migrations yourself.

Migration guides are mechanical, not literary

A migration guide that says "update your values accordingly" is not a migration guide.

Give the exact diff:

 # values.yaml - migrating platform-service 2.x -> 3.x
-resources:
-  cpu: 500m
-  memory: 512Mi
+resources:
+  requests: { cpu: "500m", memory: "512Mi" }
+  limits: { memory: "512Mi" }
-image: registry.example.com/checkout:v1.4.2
+image:
+  repository: registry.example.com/checkout
+  digest: sha256:7d3f...c9

Better: ship a codemod. A 40-line script that rewrites values files across every repo turns a quarter-long migration into an afternoon of PR review.

Best: open the PRs yourself. The platform team that sends the migration PR gets adoption; the one that files tickets gets ignored.

Escape hatches keep the path honest

Every golden path meets a workload it does not fit - a GPU job, a StatefulSet with odd storage, a vendor Helm chart.

Document the exit. A team that needs off-path files a short ADR naming the constraint and the owner, and gets an exception label that admission honors.

Undocumented escape is the failure mode. If leaving the path is easier than filing the ADR, your inventory is fiction.

Count exceptions quarterly. Three teams escaping for the same reason is a feature request, not three exceptions.

Gotchas

The path is a wiki page. Copy-paste templates have no version, so you cannot answer "who is affected?" Ship a chart or a Kustomize base with a version label.

Enforcement without a path. Turning on strict admission before the template exists just teaches teams that the platform blocks work. Ship the paved road first, then close the dirt one.

The template becomes a config language. When the chart has 200 values and conditionals nested four deep, it is no longer opinionated. Fork into two focused charts instead of one universal one.

No drift telemetry. Without the template-version label you learn about stragglers when you delete the old chart. Label everything the template renders.

Deprecating during a freeze. Check the org calendar. Enforcing removal in the middle of a code freeze earns you a permanent exception.

Alternatives

Kustomize bases over Helm charts. Simpler mental model and no templating language, and kustomize is built into kubectl. Weaker at versioning - you pin a Git ref rather than a SemVer artifact, so drift queries take more work.

cdk8s or Pulumi. Real typed languages generating manifests, so the schema is compile-checked. Good when your teams already share one language; a tax when they do not.

Crossplane or an internal operator. Teams submit a small CRD (kind: Service) and the platform reconciles everything. Strongest guardrails and the cleanest developer surface, but you now own a controller and its upgrade path.

Backstage software templates. Excellent for day-one scaffolding, weak for day-500 governance - scaffolded code is a copy, so it drifts immediately. Pair it with a versioned chart rather than using it alone.

FAQs

How many golden paths should we have? Fewer than you think. One per workload archetype - stateless HTTP, batch job, stateful store - and no more until a fourth archetype clearly exists.

What if a team refuses to migrate? Escalate with data, not opinion. "This chart version fails admission on 2026-09-01, and you have 14 workloads on it" is a schedule; "please upgrade" is a wish.

Should the platform team own the app's values.yaml? No. Own the template and the defaults; the team owns their inputs. Owning both means you own their outage.

How do we handle emergency exceptions? Time-box them. An exception label with an expiry annotation that a controller reports on weekly beats an exception that lives forever.

Do we version the admission policies too? Yes, and lag them behind the chart. Policies should enforce what the current template already produces, never what the next one will.

Is a golden path just a paved road? Same idea, different emphasis. Paved road stresses the ease of the default; golden path stresses that it is supported and versioned.

Related

• Leading a Platform, Not Just Running It
• Tech Lead Basics on Platforms
• Manifest Review Standards
• Build vs Buy Add-Ons
• Technical Leadership Best Practices

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).