The Three Pillars of Cluster Observability

Summary

Observability is the ability to answer new questions about a running system from the data it already emits, without shipping new code first.
The three pillars are metrics (aggregatable numbers over time), logs (discrete timestamped events), and traces (the path of one request across services).
Insight: Monitoring tells you a known thing broke; observability lets you investigate an unknown failure you never predicted.
Key Concepts: cardinality, golden signals (latency, traffic, errors, saturation), correlation via shared labels and trace IDs, retention cost.
When to Use: Every production Kubernetes cluster needs all three pillars; the mix shifts with scale and blast radius.
Limitations/Trade-offs: Each pillar has a different cost curve. High-cardinality metrics and full-fidelity traces get expensive fast.
Related Topics: metrics-server, Prometheus/Grafana, log aggregation, OpenTelemetry.

Foundations

A Kubernetes cluster is a distributed system where pods move, nodes fail, and requests cross many services. You cannot SSH into every container and read its state.

Instead you rely on telemetry: the data a system emits about itself. Observability is how well that telemetry lets you reason about internal state from the outside.

The classic framing splits telemetry into three signal types, each answering a different question.

Metrics answer "how much" and "how many" over time. They are numeric samples with labels, cheap to store because many events collapse into one counter or gauge.

Logs answer "what exactly happened" for a single event. They are verbose and high-fidelity but expensive to index and query at scale.

Traces answer "where did the time go" for one request as it fans out across services. A trace is a tree of spans, each span a timed unit of work.

No single pillar is sufficient alone. Metrics show that p99 latency spiked, logs show the error message, and traces show which downstream call was slow.

Metrics in Kubernetes flow along two paths. The resource metrics pipeline (metrics-server) feeds CPU and memory to the HPA and kubectl top. The full metrics pipeline (Prometheus) scrapes /metrics endpoints and stores time series for dashboards and alerts.

Prometheus stores each series as a unique combination of a metric name plus labels. Every distinct label set is a new series, so cardinality is the dominant cost driver.

Putting a user ID or a raw URL in a label explodes cardinality and can crash Prometheus. Keep labels bounded: route templates, status codes, and pod names, not unbounded identifiers.

Logs in Kubernetes start as a container's stdout and stderr. The kubelet writes those streams to files on the node under /var/log/containers/.

A DaemonSet log agent (Fluent Bit, Promtail, Vector) runs one pod per node, tails those files, enriches each line with pod and namespace labels, and ships them to a backend like Loki or Elasticsearch.

Traces require application cooperation. Code instrumented with the OpenTelemetry SDK emits spans; a context propagation header (traceparent) carries the trace ID across service boundaries.

An OpenTelemetry Collector receives, batches, and exports those spans to a backend such as Tempo or Jaeger. The collector decouples your apps from any specific vendor.

The pillars connect through shared identity. A structured log line carrying a trace_id field can be joined to its trace; an exemplar attaches a trace ID to a metric bucket. This correlation is what turns three data stores into one investigation.

Advanced Considerations & Applications

The golden signals are the highest-leverage metrics for any service: latency, traffic, errors, and saturation. Google's SRE practice recommends alerting on these rather than on low-level machine stats.

Latency should be measured as a distribution, not an average. An average of 50ms can hide that 1 percent of users wait 5 seconds, which is exactly the population most likely to churn.

Separate successful latency from failed latency. A fast stream of errors can look healthy on a latency chart while the service is entirely broken.

Cost forces prioritization. Metrics are cheapest per unit of insight, so build a broad metrics base first, add structured logs second, and add traces where request flows are complex.

Sampling manages trace cost. Head sampling decides at the start of a request; tail sampling in the collector keeps all traces that errored or were slow while dropping most fast successes.

Retention is a policy decision, not a default. Keep raw high-resolution data for days, downsample to lower resolution for months, and never keep everything forever.

At platform scale you standardize. A golden telemetry contract - every workload exposes /metrics, logs JSON to stdout, and propagates trace context - is what makes a fleet of teams observable without per-team heroics.

Common Misconceptions

"Observability is just monitoring with a new name." Monitoring watches predefined dashboards for known failures. Observability lets you ask questions you did not anticipate when you built the system.

"More logs mean better observability." Unstructured, unindexed log volume mostly raises cost and noise. A few well-structured fields and good metrics beat a firehose of text.

"Docker Engine collects my container logs in the cluster." On modern Kubernetes nodes the runtime is containerd via CRI, not Docker. The kubelet and your DaemonSet agent handle log collection.

"Averages are good enough for latency." Averages hide the tail where real users suffer. Always track percentiles like p95 and p99.

"Tracing replaces logging." Traces show request flow and timing but rarely carry full error detail. You still need logs for the specifics, ideally correlated by trace ID.

FAQs

Do I need all three pillars from day one? Start with metrics for the golden signals, add structured logs quickly, and introduce tracing once you have several services calling each other.

What is cardinality and why does it matter? Cardinality is the number of unique label combinations. High cardinality multiplies storage and memory in Prometheus and is the most common cause of an overloaded metrics stack.

Where do container logs come from in Kubernetes? From each container's stdout and stderr, written by the kubelet to node files, then tailed by a DaemonSet agent and shipped to a log backend.

Is OpenTelemetry only for traces? No. OpenTelemetry is a unified standard for traces, metrics, and logs, though its tracing support is the most mature and widely adopted piece.

How do I connect a metric spike to a specific error? Use shared labels and trace IDs. Exemplars link a metric bucket to a trace, and structured logs carrying a trace_id join logs to that same request.

Which pillar should I cut first under budget pressure? Usually traces via aggressive tail sampling, since metrics give the most coverage per dollar and logs hold the diagnostic detail.

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).

The Three Pillars of Cluster Observability

Summary

Observability is the ability to answer new questions about a running system from the data it already emits, without shipping new code first.
The three pillars are metrics (aggregatable numbers over time), logs (discrete timestamped events), and traces (the path of one request across services).
Insight: Monitoring tells you a known thing broke; observability lets you investigate an unknown failure you never predicted.
Key Concepts: cardinality, golden signals (latency, traffic, errors, saturation), correlation via shared labels and trace IDs, retention cost.
When to Use: Every production Kubernetes cluster needs all three pillars; the mix shifts with scale and blast radius.
Limitations/Trade-offs: Each pillar has a different cost curve. High-cardinality metrics and full-fidelity traces get expensive fast.
Related Topics: metrics-server, Prometheus/Grafana, log aggregation, OpenTelemetry.

Foundations

A Kubernetes cluster is a distributed system where pods move, nodes fail, and requests cross many services. You cannot SSH into every container and read its state.

Instead you rely on telemetry: the data a system emits about itself. Observability is how well that telemetry lets you reason about internal state from the outside.

The classic framing splits telemetry into three signal types, each answering a different question.

Metrics answer "how much" and "how many" over time. They are numeric samples with labels, cheap to store because many events collapse into one counter or gauge.

Logs answer "what exactly happened" for a single event. They are verbose and high-fidelity but expensive to index and query at scale.

Traces answer "where did the time go" for one request as it fans out across services. A trace is a tree of spans, each span a timed unit of work.

No single pillar is sufficient alone. Metrics show that p99 latency spiked, logs show the error message, and traces show which downstream call was slow.

Mechanics & Interactions

Prometheus stores each series as a unique combination of a metric name plus labels. Every distinct label set is a new series, so cardinality is the dominant cost driver.

Putting a user ID or a raw URL in a label explodes cardinality and can crash Prometheus. Keep labels bounded: route templates, status codes, and pod names, not unbounded identifiers.

Logs in Kubernetes start as a container's stdout and stderr. The kubelet writes those streams to files on the node under /var/log/containers/.

Traces require application cooperation. Code instrumented with the OpenTelemetry SDK emits spans; a context propagation header (traceparent) carries the trace ID across service boundaries.

An OpenTelemetry Collector receives, batches, and exports those spans to a backend such as Tempo or Jaeger. The collector decouples your apps from any specific vendor.

Advanced Considerations & Applications

Latency should be measured as a distribution, not an average. An average of 50ms can hide that 1 percent of users wait 5 seconds, which is exactly the population most likely to churn.

Separate successful latency from failed latency. A fast stream of errors can look healthy on a latency chart while the service is entirely broken.

Cost forces prioritization. Metrics are cheapest per unit of insight, so build a broad metrics base first, add structured logs second, and add traces where request flows are complex.

Sampling manages trace cost. Head sampling decides at the start of a request; tail sampling in the collector keeps all traces that errored or were slow while dropping most fast successes.

Retention is a policy decision, not a default. Keep raw high-resolution data for days, downsample to lower resolution for months, and never keep everything forever.

Common Misconceptions

"More logs mean better observability." Unstructured, unindexed log volume mostly raises cost and noise. A few well-structured fields and good metrics beat a firehose of text.

"Averages are good enough for latency." Averages hide the tail where real users suffer. Always track percentiles like p95 and p99.

"Tracing replaces logging." Traces show request flow and timing but rarely carry full error detail. You still need logs for the specifics, ideally correlated by trace ID.

FAQs

Do I need all three pillars from day one? Start with metrics for the golden signals, add structured logs quickly, and introduce tracing once you have several services calling each other.

Where do container logs come from in Kubernetes? From each container's stdout and stderr, written by the kubelet to node files, then tailed by a DaemonSet agent and shipped to a log backend.

Is OpenTelemetry only for traces? No. OpenTelemetry is a unified standard for traces, metrics, and logs, though its tracing support is the most mature and widely adopted piece.

Which pillar should I cut first under budget pressure? Usually traces via aggressive tail sampling, since metrics give the most coverage per dollar and logs hold the diagnostic detail.

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).

The Three Pillars of Cluster Observability

Search Documentation

The Three Pillars of Cluster Observability