Observability Best Practices

A short, opinionated set of practices for making a Kubernetes platform observable without drowning in cost or noise. These are the habits that separate a fleet you can debug from one you can only watch.

How to Use This List

Treat this as a platform contract, not a per-team wish list. The highest-leverage move is standardizing what every workload emits so no team has to reinvent observability.

Adopt the groups roughly in order: instrument first, then control cost, then correlate, then alert and operate. Revisit the list during design reviews and incident retrospectives.

A - Instrument Every Workload

Expose a metrics contract. Every workload serves Prometheus /metrics or emits OTLP, so discovery and scraping are uniform across the fleet.
Emit the golden signals. Export latency (as a histogram), traffic, errors, and saturation for each service, since these predict user pain better than machine stats.
Log JSON to stdout. Applications write structured logs to stdout and never manage files or shippers; the platform owns collection.
Propagate trace context. Pass the traceparent header through every service hop so a request can be followed end to end.
Add readiness and liveness probes. Turn app health into platform-actionable signals, keeping readiness and liveness separate.

B - Control Cardinality and Cost

Keep labels bounded. Never put user IDs, full URLs, or timestamps in metric or log labels; use route templates and status codes instead.
Push high-cardinality values into the body. In Loki and similar stores, keep IDs in the log body, not the index, to avoid index blowup.
Use recording rules. Precompute expensive PromQL once and let dashboards and alerts read the cheap derived series.
Sample traces at the tail. Keep all errored and slow traces while dropping most fast successes, and do it on the gateway tier.
Tier retention. Keep recent data hot for days, downsample or archive to cheap object storage, and never keep everything forever.

C - Standardize Collection

Prefer DaemonSet agents. Run one log or telemetry agent per node rather than a sidecar in every pod, unless an app truly cannot log to stdout.
Enrich with Kubernetes metadata. Attach namespace, pod, and node labels at collection time via the k8sattributes processor or the agent's Kubernetes filter.
Route telemetry through a Collector. Send app telemetry to an OpenTelemetry Collector so backends stay swappable and batching and sampling are centralized.
Remember the runtime boundary. Nodes run containers via containerd through CRI; parse the CRI log format and never assume Docker Engine collects in-cluster logs.
Manage config as CRDs. Use ServiceMonitor, PodMonitor, PrometheusRule, and OpenTelemetryCollector objects instead of hand-edited config files.

D - Correlate the Three Pillars

Carry a shared trace ID. Include trace_id in structured logs so you can pivot from a log line to its trace and back.
Attach exemplars to metrics. Link a latency histogram bucket to a sample trace so a metric spike jumps straight to a slow request.
Use consistent labels everywhere. Reuse the same app, namespace, and service labels across metrics, logs, and traces so joins work.
Build one investigation surface. Wire Grafana to metrics, logs, and traces together so an on-call engineer does not context-switch across tools.

E - Alert and Operate on SLOs

Alert on symptoms, not causes. Page on user-facing golden signals like error ratio and p99 latency rather than raw CPU or disk.
Separate failed from successful latency. A fast stream of errors can look healthy on a latency chart; split the two.
Add a for duration. Require an alert condition to persist before paging, so brief blips do not wake anyone.
Route by severity. Send severity=page to on-call and everything else to a channel, with grouping and inhibition in Alertmanager.
Autoscale on the right signal. Drive the HPA with request rate or queue depth via the Prometheus Adapter when CPU is not the real bottleneck.

When You Are Done

You should be able to answer three questions for any service in minutes: is it healthy now, what changed, and where is the time going. If a new failure mode leaves you blind, an instrumentation or correlation gap is the fix.

Review these practices when onboarding a new team, during design review, and after every incident. Observability is a standard you maintain, not a project you finish.

FAQs

Where do I start if I have nothing? Instrument the golden signals as metrics and get structured logs flowing to a central backend. Add tracing once several services call each other.

What is the single biggest cost trap? Cardinality. Unbounded labels multiply Prometheus series and Loki index size faster than any other factor.

Should every team run its own stack? No. Provide a shared platform stack and a clear emit-this contract so teams instrument once against common tooling.

Do I alert on CPU? Rarely as a page. Alert on user-facing symptoms and keep CPU for capacity and autoscaling decisions.

How do I keep traces affordable? Use tail sampling on the gateway tier to retain errored and slow traces while dropping most fast successes.

Why insist on JSON logs to stdout? It keeps apps decoupled from the log backend, makes lines queryable by field, and lets the platform own collection.

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).

How to Use This List

Treat this as a platform contract, not a per-team wish list. The highest-leverage move is standardizing what every workload emits so no team has to reinvent observability.

Adopt the groups roughly in order: instrument first, then control cost, then correlate, then alert and operate. Revisit the list during design reviews and incident retrospectives.

A - Instrument Every Workload

Expose a metrics contract. Every workload serves Prometheus /metrics or emits OTLP, so discovery and scraping are uniform across the fleet.
Emit the golden signals. Export latency (as a histogram), traffic, errors, and saturation for each service, since these predict user pain better than machine stats.
Log JSON to stdout. Applications write structured logs to stdout and never manage files or shippers; the platform owns collection.
Propagate trace context. Pass the traceparent header through every service hop so a request can be followed end to end.
Add readiness and liveness probes. Turn app health into platform-actionable signals, keeping readiness and liveness separate.

B - Control Cardinality and Cost

Keep labels bounded. Never put user IDs, full URLs, or timestamps in metric or log labels; use route templates and status codes instead.
Push high-cardinality values into the body. In Loki and similar stores, keep IDs in the log body, not the index, to avoid index blowup.
Use recording rules. Precompute expensive PromQL once and let dashboards and alerts read the cheap derived series.
Sample traces at the tail. Keep all errored and slow traces while dropping most fast successes, and do it on the gateway tier.
Tier retention. Keep recent data hot for days, downsample or archive to cheap object storage, and never keep everything forever.

C - Standardize Collection

Prefer DaemonSet agents. Run one log or telemetry agent per node rather than a sidecar in every pod, unless an app truly cannot log to stdout.
Enrich with Kubernetes metadata. Attach namespace, pod, and node labels at collection time via the k8sattributes processor or the agent's Kubernetes filter.
Route telemetry through a Collector. Send app telemetry to an OpenTelemetry Collector so backends stay swappable and batching and sampling are centralized.
Remember the runtime boundary. Nodes run containers via containerd through CRI; parse the CRI log format and never assume Docker Engine collects in-cluster logs.
Manage config as CRDs. Use ServiceMonitor, PodMonitor, PrometheusRule, and OpenTelemetryCollector objects instead of hand-edited config files.

D - Correlate the Three Pillars

Carry a shared trace ID. Include trace_id in structured logs so you can pivot from a log line to its trace and back.
Attach exemplars to metrics. Link a latency histogram bucket to a sample trace so a metric spike jumps straight to a slow request.
Use consistent labels everywhere. Reuse the same app, namespace, and service labels across metrics, logs, and traces so joins work.
Build one investigation surface. Wire Grafana to metrics, logs, and traces together so an on-call engineer does not context-switch across tools.

E - Alert and Operate on SLOs

Alert on symptoms, not causes. Page on user-facing golden signals like error ratio and p99 latency rather than raw CPU or disk.
Separate failed from successful latency. A fast stream of errors can look healthy on a latency chart; split the two.
Add a for duration. Require an alert condition to persist before paging, so brief blips do not wake anyone.
Route by severity. Send severity=page to on-call and everything else to a channel, with grouping and inhibition in Alertmanager.
Autoscale on the right signal. Drive the HPA with request rate or queue depth via the Prometheus Adapter when CPU is not the real bottleneck.

When You Are Done

Review these practices when onboarding a new team, during design review, and after every incident. Observability is a standard you maintain, not a project you finish.

FAQs

Where do I start if I have nothing? Instrument the golden signals as metrics and get structured logs flowing to a central backend. Add tracing once several services call each other.

What is the single biggest cost trap? Cardinality. Unbounded labels multiply Prometheus series and Loki index size faster than any other factor.

Should every team run its own stack? No. Provide a shared platform stack and a clear emit-this contract so teams instrument once against common tooling.

Do I alert on CPU? Rarely as a page. Alert on user-facing symptoms and keep CPU for capacity and autoscaling decisions.

How do I keep traces affordable? Use tail sampling on the gateway tier to retain errored and slow traces while dropping most fast successes.

Why insist on JSON logs to stdout? It keeps apps decoupled from the log backend, makes lines queryable by field, and lets the platform own collection.

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).

Observability Best Practices

Search Documentation

Observability Best Practices