Log Aggregation
Summary
- Log aggregation collects each container's stdout and stderr from every node and ships it to a central, queryable backend.
- The standard pattern is a DaemonSet agent (Fluent Bit, Vector, Promtail) that tails node log files and enriches lines with Kubernetes metadata.
- Insight: Applications should log JSON to stdout and never manage log files or shippers themselves. The platform owns collection.
- Key Concepts: stdout/stderr contract, DaemonSet agent, structured JSON logs, label vs index cardinality, retention tiers.
- When to Use: Every cluster needs central logs for debugging, audit, and correlation with metrics and traces.
- Limitations: Logs are the most expensive pillar per byte; volume, indexing, and retention must be controlled.
Recipe
Have apps log JSON to stdout, then run a node-level agent to collect and ship it. With Loki the fastest path is the Helm chart plus its Promtail or Grafana Alloy agent.
helm repo add grafana https://grafana.github.io/helm-charts
helm install loki grafana/loki-stack -n logging --create-namespaceThis deploys Loki plus a DaemonSet agent on every node, and Grafana can then query logs by label.
Working Example
First, emit structured logs from the application to stdout.
{"ts":"2026-07-15T10:00:00Z","level":"error","msg":"payment declined","route":"/api/checkout","status":402,"order_id":"o-991","trace_id":"a1b2c3d4"}Run a Fluent Bit DaemonSet that tails container logs and enriches them.
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: fluent-bit
namespace: logging
spec:
selector:
matchLabels:
app: fluent-bit
template:
metadata:
labels:
app: fluent-bit
spec:
serviceAccountName: fluent-bit
containers:
- name: fluent-bit
image: fluent/fluent-bit:latest
volumeMounts:
- name: varlog
mountPath: /var/log
readOnly: true
tolerations:
- operator: Exists
volumes:
- name: varlog
hostPath:
path: /var/logConfigure the pipeline to tail, add Kubernetes metadata, and output.
[INPUT]
Name tail
Path /var/log/containers/*.log
Parser cri
Tag kube.*
[FILTER]
Name kubernetes
Match kube.*
Merge_Log On
[OUTPUT]
Name loki
Match *
Host loki.logging.svc
Labels job=fluent-bitThe kubernetes filter attaches namespace, pod, and container labels; Merge_Log On parses the JSON app line into fields. The tolerations block ensures the agent also runs on tainted nodes like control-plane.
Deep Dive
Where logs come from
On modern nodes the runtime is containerd via CRI, not Docker Engine. The kubelet writes each container's stdout and stderr to files under /var/log/containers/, symlinked to /var/log/pods/.
The agent tails these files, so it must parse the CRI log format. The cri parser handles the timestamp and stream prefix each line carries.
DaemonSet agent vs sidecar shipping
The DaemonSet pattern runs one agent per node and collects for all pods there. It is efficient, centrally managed, and the default choice.
A sidecar shipper runs a logging container inside each pod. Use it only when an app writes to a file it cannot redirect to stdout, since it multiplies resource cost across every pod.
Never bake a log shipper into the application image. That couples the app to a backend and breaks the platform's ownership of collection.
Structured logs and cardinality
JSON logs turn free text into fields you can filter and aggregate. Filtering by status or route is trivial when those are real fields.
Backends differ on where cardinality hurts. Loki indexes only labels and keeps the log body unindexed, so labels must be low-cardinality; put high-cardinality values like order_id in the log body, not in labels.
Elasticsearch indexes fields more broadly, which is powerful but costlier at high cardinality. Match your field strategy to the backend.
Correlation and retention
A trace_id field lets you pivot from a log line to its full trace, and back from a slow trace to the exact error. This is what ties the three pillars together.
Retention is tiered on purpose. Keep recent logs hot and searchable for days, archive older logs to cheap object storage, and drop or heavily sample verbose debug logs.
Gotchas
Multiline stack traces split into many entries. Each newline becomes a separate record. Enable a multiline parser in the agent so a Java or Python traceback is one log event.
Log volume blows up storage cost. Debug logging left on in production dominates spend. Set levels per environment and sample high-volume, low-value lines.
High-cardinality labels break Loki. Putting user or order IDs in labels explodes the index. Keep those values in the log body and reserve labels for bounded dimensions.
Agent misses logs on some nodes. The DaemonSet lacks tolerations for tainted nodes, so control-plane or GPU nodes are skipped. Add tolerations so it runs everywhere.
App logs to a file, not stdout. A container writing to a local file bypasses the kubelet stream and the agent. Redirect to stdout, or as a last resort add a sidecar.
Alternatives
Loki indexes labels only and stores the body cheaply, pairing naturally with Grafana. Choose it for cost-efficient, label-driven logging alongside your metrics.
Elasticsearch or OpenSearch with Kibana gives rich full-text search and analytics. Pick it when powerful ad hoc querying justifies higher storage and operational cost.
Vector is a high-performance agent and router that can transform and fan out to multiple sinks. Use it when you need heavy in-flight processing or multi-backend delivery.
OpenTelemetry Collector can also collect logs and unify them with traces and metrics under one agent. Prefer it when standardizing all telemetry on OTel.
FAQs
Should apps write log files? No. Log JSON to stdout and let the platform's DaemonSet agent collect it. Managing files couples the app to infrastructure.
Does Docker collect my cluster logs? No. Nodes run containers with containerd via CRI. The kubelet writes log files and a DaemonSet agent ships them.
DaemonSet or sidecar? Prefer a DaemonSet agent for efficiency and central control. Use a sidecar only when an app cannot log to stdout.
Why is my Loki index huge? You put high-cardinality values in labels. Move IDs into the log body and keep labels bounded.
How do I connect a log to a trace? Include a trace_id field in every structured log line, then pivot on it in Grafana between Loki and the trace backend.
How do I control logging cost? Set log levels per environment, sample verbose lines, and tier retention with cheap object storage for archives.
Related
- The Three Pillars of Cluster Observability
- Observability Basics
- Prometheus & Grafana
- OpenTelemetry Collectors
- Observability Best Practices
Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).