Defect Scenarios Best Practices

Every incident in this section shares a shape: a plausible setting meets a trigger and becomes an outage.

The highest-leverage response is to convert each lesson into a machine-enforced rule so the same defect cannot reach the cluster again.

How to Use This List

Treat each practice as a candidate admission policy, a CI check, or a default in your base manifests.

Adopt the group that matches your current pain first, then work toward enforcing all of them.

Enforcement beats documentation: a rule in a mutating or validating policy stops the defect at apply time, while a wiki page does not.

A - Pin and verify images

• Ban the :latest tag in production. Enforce with a Kyverno or Validating Admission Policy rule so a mutable tag cannot be applied.
• Prefer digests for critical workloads. Reference image@sha256:... so the running bytes are cryptographically fixed and rollback is exact.
• Set imagePullPolicy explicitly. Avoid the implicit Always that :latest triggers, which maximizes drift across nodes.
• Sign and verify images. Use cosign and require a valid signature at admission so only vetted digests run.
• Scan before deploy. Gate the pipeline on Trivy or Grype so known-vulnerable images fail CI, not production.

B - Size resources honestly

• Always set memory requests and limits. Requests drive scheduling and QoS; limits are the kernel-enforced ceiling that prevents a noisy neighbor.
• Make runtimes cgroup-aware. Use -XX:MaxRAMPercentage for the JVM and --max-old-space-size for Node so the heap fits under the limit.
• Leave off-heap headroom. Keep the limit above the heap for metaspace, threads, and native buffers so peaks do not OOMKill.
• Right-size from data, not guesses. Use VPA recommendations and kubectl top to set requests and limits from observed peak usage.
• Enforce limits at admission. Reject containers with no memory limit so unbounded pods cannot land on shared nodes.

C - Configure probes for reality

• Use a startupProbe for slow boot. It suspends liveness during startup so a cold start under load is not mistaken for a deadlock.
• Keep liveness cheap and deadlock-only. Never make liveness depend on a database, or a dependency blip will restart every pod at once.
• Put dependency checks in readiness. Readiness sheds traffic without killing the pod, which is the correct pressure valve under load.
• Set thresholds from p99 latency. Choose timeoutSeconds above measured peak latency so a busy-but-healthy pod is not killed.
• Separate /livez and /readyz endpoints. Distinct handlers keep restart logic and traffic logic from coupling.

D - Contain the blast radius

• Apply default-deny NetworkPolicy. Deny ingress by default per namespace and allow only the flows each service needs.
• Set ResourceQuota and LimitRange. Bound what one namespace can consume and supply default limits so unset pods still get a ceiling.
• Stage rollouts conservatively. Use small maxUnavailable, a PodDisruptionBudget, and canary or progressive delivery to slow amplification.
• Scope admission webhooks. Use namespaceSelector and prefer failurePolicy: Ignore for non-security webhooks so one webhook cannot wedge the cluster.
• Enforce Pod Security Standards (restricted). Reject root, privilege escalation, and host namespaces so a compromise cannot spread.

E - Detect defects before users do

• Alert on restart and OOMKill counts. A climbing restartCount or Reason: OOMKilled is signal that green request dashboards will miss.
• Alert on rollout stalls and probe failures. Catch drift and crash loops from control-loop events, not only from error rates.
• Compare image digests across pods. Watch for more than one imageID per Deployment to catch silent version drift.
• Run node-drain and load drills. Exercise the triggers - scaling, eviction, dependency slowdown - so latent faults surface in staging.
• Turn each postmortem into a policy. Every incident should ship a new admission rule or CI check so the class of defect cannot recur.

When You Are Done

You should be able to point at a policy, check, or default for every scenario in this section.

If a defect can still be applied to the cluster by hand, it is documentation, not enforcement, and it will happen again.

Aim for a state where the plausible mistakes in this section fail loudly at apply time or in CI, long before a trigger can reach production.

FAQs

Where should I start if I can only do one thing? Ban :latest and require memory limits at admission - those two rules eliminate the two most common silent defects.

Policy engine or Validating Admission Policy? Validating Admission Policy (CEL, in-tree) covers many checks with no extra controller. Kyverno or Gatekeeper add mutation and richer libraries when you need them.

Will strict policies block legitimate deploys? Roll out in audit or warn mode first, fix violations, then switch to enforce so teams migrate without surprise outages.

Do these rules replace load testing? No. Policies stop known-bad patterns, but only load and chaos testing reveal the trigger thresholds behind sizing and probe settings.

How do I keep rules from drifting? Store policies in Git, deploy them via GitOps, and test them in CI so the guardrails are themselves versioned and reviewed.

Related

• How Misconfiguration Becomes Outage
• Defect Scenarios Basics
• :latest in Production
• Probe Misconfiguration
• Resource Limit OOMKilled

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).