metrics-server & HPA

Summary

• metrics-server is a cluster add-on that collects CPU and memory usage from each kubelet and serves it through the Metrics API (metrics.k8s.io).
• It powers kubectl top and the CPU/memory targets of the HorizontalPodAutoscaler (HPA).
• Insight: metrics-server is a short-lived in-memory pipeline, not a monitoring system. It holds only the latest sample and keeps no history.
• Key Concepts: resource metrics pipeline, Metrics API, HPA v2, target utilization, stabilization window.
• When to Use: Install it on every cluster that runs autoscaling or wants kubectl top. Use Prometheus Adapter for custom or external metrics.
• Limitations: No historical data, not meant for monitoring, and it needs correct pod resource requests to compute utilization.

Recipe

Install metrics-server, confirm the Metrics API responds, then attach an HPA that scales on CPU.

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
kubectl top nodes
kubectl top pods -A

If kubectl top returns data, the resource metrics pipeline is working and the HPA can consume it.

Working Example

Start with a Deployment that declares CPU requests, since the HPA computes utilization as a percentage of the request.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  namespace: web
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: web
          image: registry.example.com/web@sha256:abc123
          resources:
            requests:
              cpu: 200m
              memory: 256Mi
            limits:
              memory: 512Mi

Now add an autoscaler targeting 70 percent average CPU utilization.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: web
  namespace: web
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300

Watch the HPA make decisions in real time.

kubectl get hpa web -n web --watch

The TARGETS column shows current versus target utilization, and REPLICAS moves as load changes.

Deep Dive

How the resource metrics pipeline works

Each kubelet exposes a /metrics/resource endpoint sourced from the container runtime through CRI. metrics-server scrapes every kubelet on an interval, aggregates the samples, and serves them via the aggregated API server.

Because it keeps only the most recent scrape in memory, metrics-server is cheap and fast but has no query language and no retention.

How the HPA computes replicas

The HPA controller runs a reconcile loop, by default every 15 seconds. It reads current utilization from the Metrics API and applies a ratio formula.

Desired replicas equal current replicas times current metric value divided by target value, rounded up. If two pods average 140 percent against a 70 percent target, the HPA doubles to four replicas.

Utilization is always a fraction of the pod's request. Without CPU requests set, the HPA cannot compute a percentage and reports <unknown> targets.

Scaling behavior and stability

The behavior block controls how aggressively the HPA reacts. A scaleDown stabilization window of 300 seconds makes the controller pick the highest recommendation over the last five minutes, preventing rapid flapping when traffic is spiky.

You can also cap scale-up rate with policies, for example allowing at most a percentage or a fixed pod count per period.

Custom and external metrics

CPU and memory are often the wrong scaling signal. Request rate or queue depth track load more directly.

The Prometheus Adapter implements the custom.metrics.k8s.io and external.metrics.k8s.io APIs, letting an HPA scale on a Prometheus query such as requests per second per pod. metrics-server stays responsible only for CPU and memory.

Gotchas

HPA shows <unknown> for targets. The pods lack resource requests, or metrics-server is not running. Set CPU requests and confirm kubectl top pods works.

metrics-server pod crashes with TLS errors. On some clusters the kubelet serving certificate is not signed by the cluster CA. The documented fix is the --kubelet-insecure-tls flag in non-production, or provisioning proper kubelet certificates in production.

Autoscaler flaps up and down. Add a scaleDown stabilization window and scale-up policies so short spikes do not thrash replica count.

Scaling on CPU misses the real bottleneck. If your service is I/O or queue bound, CPU stays low while latency climbs. Scale on a custom metric via the Prometheus Adapter instead.

HPA and VPA both changing CPU. Running the HorizontalPodAutoscaler and VerticalPodAutoscaler on the same CPU resource conflicts. Keep the VPA off any resource the HPA already targets.

Alternatives

KEDA extends autoscaling to event sources like Kafka lag, queue length, or cron schedules, and can scale to zero. Choose it when scaling on external event backlogs rather than CPU.

Cluster Autoscaler or Karpenter add and remove nodes when pods cannot be scheduled. They complement the HPA, which scales pods; you usually run both.

VerticalPodAutoscaler right-sizes requests and limits over time instead of changing replica count. Use it for workloads that do not parallelize well.

Prometheus Adapter with HPA is the standard path when you need request-rate or business-metric scaling while keeping the native HPA controller.

FAQs

Is metrics-server a monitoring tool? No. It holds only the latest sample and has no history or alerting. Use Prometheus for monitoring and metrics-server for autoscaling and kubectl top.

Why does my HPA report unknown CPU? The target pods have no CPU requests, so utilization cannot be computed as a percentage. Add resources.requests.cpu.

Can one HPA use multiple metrics? Yes. List several entries under metrics; the HPA computes a recommendation per metric and uses the largest.

Does metrics-server run the pods' containers? No. Nodes run containers via containerd through CRI. metrics-server only reads usage that the kubelet exposes.

How do I scale on requests per second? Install the Prometheus Adapter, expose the query as a custom metric, and reference it in the HPA's metrics list.

How fast does the HPA react? The controller reconciles about every 15 seconds, but scale-down honors the stabilization window, so downscaling is deliberately slower than upscaling.

Related

• The Three Pillars of Cluster Observability
• Observability Basics
• Prometheus & Grafana
• OpenTelemetry Collectors
• Observability Best Practices

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).