Defect Scenarios Basics

This page shows the small, plausible container and Kubernetes settings that turn into outages, with a runnable snippet and the fix for each.

Work through these to build intuition before the deeper incident walkthroughs in this section.

Prerequisites

• Kubernetes 1.36.2 cluster access with kubectl configured (kind or a cloud cluster both work).
• Docker Engine 29.6.1 with BuildKit for local image builds.
• Helm 3 and Compose v2 optional for later examples.
• A test namespace so experiments cannot touch shared workloads.

kubectl create namespace defect-lab
kubectl config set-context --current --namespace=defect-lab

Basic Examples

1. The mutable tag that drifts

Pulling :latest means two identical manifests can run different code.

containers:
  - name: api
    image: registry.example.com/api:latest   # today's latest, not yesterday's

• The tag is a moving pointer, so a rebuild changes what runs without a manifest change.
• A new pod on a scaled-up node can pull a different digest than its siblings.
• Fix: pin an immutable tag or a digest, image: registry.example.com/api@sha256:....
• Set imagePullPolicy: IfNotPresent with a fixed tag so nodes do not silently re-pull.

2. Liveness probe that starts too soon

A liveness probe that fires before the app is ready restarts a healthy container.

livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 1
  periodSeconds: 5

• initialDelaySeconds: 1 assumes a one-second startup that does not survive load.
• Three failed checks trigger a kubelet restart, adding another cold start.
• Fix: use a startupProbe for slow boot, then let liveness take over.
• Reserve liveness for true deadlocks, not for slow-but-progressing startup.

3. Missing memory limit

A pod with no memory limit can consume a node and get OOMKilled unpredictably.

resources:
  requests:
    memory: "256Mi"
  limits:
    memory: "512Mi"

• Without limits, the kernel OOM killer, not Kubernetes, decides who dies.
• Without requests, the scheduler cannot bin-pack safely and overcommits the node.
• Fix: always set both requests and limits for memory.
• Watch kubectl get pod RESTARTS and describe for Reason: OOMKilled.

4. Runtime confusion: Docker is not your node runtime

Build with Docker, but pods run on containerd via the CRI - dockershim is gone.

# On a node, inspect running containers with the CRI tool, not docker
crictl ps
crictl images

• docker ps on a node shows nothing about pods on a modern cluster.
• Use Docker and BuildKit for build and local Compose, not for in-cluster inspection.
• Fix: standardize on crictl for node-level debugging.
• Your image format (OCI) is portable, so the build tool and runtime differ by design.

5. Running as root

A container that runs as root widens the blast radius of any compromise.

securityContext:
  runAsNonRoot: true
  runAsUser: 1000
  allowPrivilegeEscalation: false
  capabilities:
    drop: ["ALL"]

• Many base images default to UID 0 unless you override it.
• Restricted Pod Security Standards reject root containers at admission.
• Fix: build a non-root image and set runAsNonRoot: true.
• Drop all Linux capabilities and add back only what the app needs.

6. Deploying without a readiness probe

Without readiness, a Service sends traffic to a pod that is still warming up.

readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  periodSeconds: 5

• Readiness gates whether a pod receives Service traffic, unlike liveness.
• No readiness means early requests hit an unready pod and error.
• Fix: add readiness that checks dependencies your app needs to serve.
• During rollout, readiness lets old pods drain before new ones take load.

7. Rolling everything at once

An aggressive rollout amplifies a bad config across the whole Deployment instantly.

strategy:
  rollingUpdate:
    maxUnavailable: 25%
    maxSurge: 25%

• A large maxUnavailable can drop capacity below what traffic needs.
• A defect ships to all replicas before you notice the first failing pod.
• Fix: keep surges conservative and add a PodDisruptionBudget.
• Pair rollouts with canary or progressive delivery for real safety.

Intermediate Examples

8. cgroup-aware memory for the JVM

A JVM that ignores the cgroup limit sizes its heap to the node, then gets OOMKilled.

FROM eclipse-temurin:21-jre
# Modern JVMs read cgroup limits by default; tune the fraction explicitly
ENV JAVA_OPTS="-XX:MaxRAMPercentage=75.0"
ENTRYPOINT ["sh", "-c", "java $JAVA_OPTS -jar /app/app.jar"]

• The heap plus off-heap memory must fit under the container limits.memory.
• MaxRAMPercentage sizes the heap from the detected cgroup limit, not the host.
• Verify with kubectl exec and check the reported max heap against the limit.
• Leave headroom for metaspace, threads, and native buffers.

9. Default-deny network baseline

Open pod-to-pod networking lets one compromised workload reach everything.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
spec:
  podSelector: {}
  policyTypes: ["Ingress"]

• An empty podSelector applies the policy to every pod in the namespace.
• With no ingress rules, all inbound pod traffic is denied by default.
• Add explicit allow policies for the flows each service actually needs.
• This shrinks blast radius so a defect in one service cannot pivot laterally.

10. Reproduce and confirm a defect

Before fixing, reproduce the failure so you know the trigger.

kubectl apply -f bad-liveness.yaml
kubectl get pod -w                 # watch RESTARTS climb under load
kubectl describe pod api-xxxx      # read Events for Liveness probe failed
kubectl logs api-xxxx --previous   # inspect the crashed container

• --previous reads the log of the container instance that was just killed.
• The Events section names the trigger: probe failure, OOMKilled, or ImagePullBackOff.
• Reproduce, capture the signal, then apply the fix and re-watch.
• A reliable repro turns a mystery into a one-line configuration change.

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).