Observability Basics

This page is a hands-on intro to reading a cluster's health signals with tools you already have, then wiring the golden signals every platform team relies on. Examples move from raw kubectl inspection to a first Prometheus scrape and alert.

Prerequisites

• Kubernetes 1.36.2 cluster with kubectl configured against it.
• metrics-server installed for kubectl top (see the metrics-server page).
• Helm 3 for installing the Prometheus stack.
• A workload emitting Prometheus-format metrics, or a demo app you can deploy.

Quick install of the metrics-server if your cluster lacks it:

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

Basic Examples

1. Inspect pod state and restarts

The first observability tool is the object status the API server already tracks.

kubectl get pods -n web -o wide

• STATUS shows Running, CrashLoopBackOff, Pending, or OOMKilled hints.
• The RESTARTS column with a recent timestamp signals a crash loop.
• -o wide adds the node and pod IP, useful for correlating with node problems.
• No agent is required; this data comes straight from the control plane.

2. Read events for a failing pod

Events explain why a pod is stuck before you reach for logs.

kubectl describe pod web-7d9f -n web

• The Events section lists scheduling, image-pull, and probe failures.
• Failed to pull image points to a registry or tag problem.
• Liveness probe failed shows the kubelet is restarting the container.
• Events expire after about an hour by default, so read them promptly.

3. Tail container logs

Container stdout and stderr are the fastest raw diagnostic.

kubectl logs -f deploy/web -n web --tail=100

• -f streams new lines; --tail limits history.
• Add --previous to read the last terminated container after a crash.
• Add -c <container> to select one container in a multi-container pod.
• These logs come from files the kubelet writes on the node.

4. See live resource usage

kubectl top reads the resource metrics pipeline for a quick saturation view.

kubectl top pods -n web --sort-by=memory

• Requires metrics-server; without it the command errors.
• Compare usage against the pod's requests and limits.
• Memory near the limit predicts an imminent OOMKilled.
• Values are recent snapshots, not historical trends.

5. Add a liveness and readiness probe

Probes turn app health into signals the platform can act on.

readinessProbe:
  httpGet:
    path: /healthz
    port: 8080
  periodSeconds: 5
livenessProbe:
  httpGet:
    path: /livez
    port: 8080
  periodSeconds: 10

• Readiness controls whether the pod receives Service traffic.
• Liveness restarts a hung container that no longer responds.
• Keep /healthz cheap so probes do not add load.
• Separate readiness from liveness so a busy pod is not killed.

6. Expose a Prometheus metrics endpoint

Most metric collection starts with an HTTP /metrics path.

apiVersion: v1
kind: Service
metadata:
  name: web
  labels:
    app: web
spec:
  selector:
    app: web
  ports:
    - name: metrics
      port: 8080

• A named metrics port lets a scraper target it cleanly.
• The app must serve Prometheus text format at /metrics.
• Consistent labels like app: web make discovery predictable.
• The port name is referenced later by a ServiceMonitor.

7. Structure logs as JSON

Structured logs are queryable; plain text is not.

{"ts":"2026-07-15T10:00:00Z","level":"error","msg":"db timeout","route":"/api/orders","status":500,"trace_id":"a1b2c3"}

• Fixed fields like level, route, and status enable filtering.
• A trace_id field lets you jump from a log to its trace.
• JSON to stdout keeps the app decoupled from the log backend.
• Avoid unbounded fields that explode index cardinality.

Intermediate Examples

8. Install the kube-prometheus-stack

One Helm chart brings Prometheus, Grafana, and Alertmanager together.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install kps prometheus-community/kube-prometheus-stack -n monitoring --create-namespace

• Installs the Prometheus Operator plus default cluster dashboards.
• ServiceMonitor and PrometheusRule CRDs become available after install.
• Grafana ships preloaded with Kubernetes dashboards.
• Override values.yaml to set retention and resource limits.

9. Scrape a workload with a ServiceMonitor

The Operator turns a CRD into scrape configuration for you.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: web
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: web
  endpoints:
    - port: metrics
      interval: 15s

• selector matches the Service labels from example 6.
• port: metrics refers to the named Service port, not a number.
• The Operator reconciles this into the Prometheus config automatically.
• Keep interval consistent to make rate calculations stable.

10. Alert on a golden signal

An error-rate alert catches real user pain, not machine noise.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: web-slo
  namespace: monitoring
spec:
  groups:
    - name: web
      rules:
        - alert: HighErrorRate
          expr: sum(rate(http_requests_total{status=~"5..",app="web"}[5m])) / sum(rate(http_requests_total{app="web"}[5m])) > 0.05
          for: 10m
          labels:
            severity: page
          annotations:
            summary: "5xx error ratio above 5% for 10m"

• The expression is the ratio of 5xx responses to all requests.
• for: 10m avoids paging on a brief blip.
• severity: page routes to Alertmanager for on-call notification.
• This alerts on the errors golden signal rather than raw CPU.

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).