Manifest Review Standards

Summary

• A manifest review standard is the published, machine-checked bar every Kubernetes YAML must clear before merge - labels, probes, resources, security context - plus the short list of judgment calls a human still has to make.
• Insight: The purpose of the standard is to make human review rarer, not more thorough. Anything you can state as a rule belongs in a policy engine, and everything left in the human's lap should be a question a machine genuinely cannot answer.
• Key Concepts: required labels (ownership + inventory), probe correctness (readiness vs liveness are different questions), resource justification (a number with a source), shift-left enforcement (CI fails before admission does), reviewer checklist (5 items, not 40).
• When to Use: As soon as more than one team writes manifests into a shared cluster, and before your first "who owns this pod?" incident.
• Limitations: A standard cannot catch a wrong architecture. It stops bad YAML, not a bad design.

Recipe

Split every rule into "a machine checks this" and "a human judges this."

1. Write the required-labels and required-fields rules as Kyverno policies in audit mode.
2. Run the same policies in CI against rendered manifests so failures land in the PR, not at deploy.
3. Flip to enforce after the audit backlog is clean.
4. Give humans a 5-item review checklist covering only the judgment calls.
5. Review the checklist quarterly - anything asked twice becomes a policy.

Working Example

The required-labels rule, enforced identically in CI and admission.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-standard-labels
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: require-ownership-labels
      match:
        any:
          - resources:
              kinds: [Deployment, StatefulSet, CronJob]
      validate:
        message: >-
          Workloads must set app.kubernetes.io/name, app.kubernetes.io/part-of,
          and example.com/owner (a Slack channel or team handle).
        pattern:
          metadata:
            labels:
              app.kubernetes.io/name: "?*"
              app.kubernetes.io/part-of: "?*"
              example.com/owner: "?*"
    - name: require-probes-and-resources
      match:
        any:
          - resources:
              kinds: [Deployment, StatefulSet]
      validate:
        message: "Containers require readinessProbe, memory requests, and memory limits."
        pattern:
          spec:
            template:
              spec:
                containers:
                  - readinessProbe:
                      periodSeconds: ">0"
                    resources:
                      requests:
                        memory: "?*"
                      limits:
                        memory: "?*"

Wire it into CI so the feedback arrives at PR time:

# ci/validate-manifests.sh
set -euo pipefail
 
helm template checkout ./charts/platform-service -f apps/checkout/values.yaml > /tmp/rendered.yaml
 
kyverno apply ./policies/ --resource /tmp/rendered.yaml --detailed-results
kubectl apply --dry-run=server -f /tmp/rendered.yaml

The --dry-run=server pass matters: it runs real admission (including webhooks and quota) without persisting, so CI and the cluster agree on what is legal.

A manifest that passes the machine, with the human-relevant parts annotated:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout
  labels:
    app.kubernetes.io/name: checkout
    app.kubernetes.io/part-of: payments
    example.com/owner: "#team-payments"
  annotations:
    example.com/resource-rationale: >-
      p99 RSS 380Mi over 30d (see grafana.example.com/d/checkout-mem);
      limit set at 512Mi for ~35% headroom. CPU request from p95 usage 420m.
spec:
  replicas: 4
  template:
    spec:
      containers:
        - name: app
          image: registry.example.com/checkout@sha256:7d3f...c9
          resources:
            requests: { cpu: "500m", memory: "512Mi" }
            limits: { memory: "512Mi" }
          readinessProbe:
            httpGet: { path: /healthz/ready, port: 8080 }
            periodSeconds: 5
            failureThreshold: 2
          livenessProbe:
            httpGet: { path: /healthz/live, port: 8080 }
            periodSeconds: 10
            failureThreshold: 6

Deep Dive

Labels earn their place by answering a question

Require a label only when you can name the query it serves.

app.kubernetes.io/name and part-of drive dashboards and service maps. example.com/owner answers the 3 a.m. question, so make it a routable Slack channel and never a person's name.

Everything else is optional until an on-call engineer asks for it. Label sprawl is real, and each required label is a tax on every manifest forever.

Readiness and liveness answer different questions

Readiness asks "should traffic go here right now?" Liveness asks "is this process unrecoverable?"

The common failure is pointing both at the same handler. When a dependency blips, readiness should fail and drain traffic - but a shared handler also fails liveness, and the kubelet restarts a healthy process, turning a brownout into a crash loop.

Liveness should check only in-process health: event loop alive, no deadlock. It must not touch the database.

Give liveness a slack failureThreshold and periodSeconds. A liveness probe that is tighter than your GC pause is a self-inflicted outage.

Use startupProbe for slow boots rather than inflating initialDelaySeconds, so a slow start does not also mean a slow crash detection later.

Resource justification is the highest-value review item

"Where did 512Mi come from?" is the question that catches the most real problems, and no policy engine can ask it.

Require a rationale annotation with a source: an observed percentile, a load-test result, or a link. "It's what the last service used" is not a source.

Memory limits should equal memory requests for anything latency-sensitive. Memory is incompressible - exceeding the limit is an OOMKill, not throttling - so a gap between request and limit is a promise the node may not keep.

CPU limits are the opposite. Setting a CPU limit invites CFS throttling at p99 even when the node is idle; most teams should set CPU requests and skip CPU limits unless they have a hard multi-tenancy reason.

Point reviewers at VPA in Off mode. Its recommendations turn the resource conversation from opinion into data.

The human checklist is short on purpose

Five items, all judgment:

1. Do the resource numbers have a stated source, and is the memory limit equal to the request?
2. Does liveness avoid checking dependencies that readiness already covers?
3. Is the blast radius right - replicas, PDB, topology spread for anything user-facing?
4. Is this the golden-path template, and if not, is there an ADR?
5. Does the rollout strategy match the workload's tolerance for two versions running at once?

A 40-item human checklist gets skimmed and rubber-stamped. Five items get read.

Gotchas

Enforce mode on day one. Turning on Enforce before auditing breaks every existing pipeline at once. Run Audit, clear the backlog, then flip.

CI and admission disagree. Linting rendered YAML with a different ruleset than the cluster runs means green PRs that fail at deploy. Run the same policy files in both, and use --dry-run=server.

Owner labels naming people. People change teams; channels outlive them. Require a channel or team handle and validate the pattern.

Probes that lie. A readiness endpoint returning 200 unconditionally passes every policy and every review while telling you nothing. Spot-check what the handler actually does.

Reviewing YAML humans do not write. If the golden-path chart renders the manifest, review the values file. Reviewing generated output wastes everyone's afternoon.

Alternatives

OPA Gatekeeper instead of Kyverno. Rego is more expressive for cross-resource logic and is the right pick if you already run OPA elsewhere. The cost is a real language to learn; Kyverno's YAML matches how most teams think about manifests.

Validating Admission Policy (CEL, in-tree). No webhook to run or keep highly available, and it is GA. Good for simple field checks; it cannot mutate or generate, so most platforms still keep a policy engine alongside it.

kubeconform / kubectl-validate in CI only. Cheap and fast for schema correctness, but CI-only enforcement is bypassable by anyone with cluster access. Fine as a first gate, never as the last one.

Datree or Polaris style scorecards. Great for a maturity dashboard and for socializing the standard. Advisory by nature - pair with real admission if the rule actually matters.

FAQs

Should the standard live in a doc or in code? In code, with the doc generated from it. Two sources of truth means the doc is wrong within a month.

How do we handle vendor Helm charts that fail policy? Namespace-scoped exclusions with an expiry, plus an upstream issue. Do not weaken the cluster-wide rule for one vendor.

Do CronJobs need probes? No - probes are for long-running pods. Match your probe policy to Deployments and StatefulSets, and check backoffLimit and activeDeadlineSeconds on jobs instead.

Is readOnlyRootFilesystem a review item or a policy? A policy. It is binary and mechanical, which is exactly what admission is for. Pod Security Standards at restricted covers most of this class.

What about resource requests for burstable batch work? Different archetype, different rule. Batch can run Burstable with a gap between request and limit; user-facing services should not.

Who arbitrates when a team disputes a rule? The platform team owns the rule, but a dispute is data. Two teams disputing the same rule usually means the rule is wrong.

Related

• Leading a Platform, Not Just Running It
• Tech Lead Basics on Platforms
• Golden Path Governance
• Build vs Buy Add-Ons
• Technical Leadership Best Practices

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).