Reference: Worker-Heavy Cluster

Summary

• A worked example of a cluster where 90% of compute is asynchronous work - queue consumers, video transcoding, and nightly batch - scaled by KEDA onto mostly spot capacity.
• Insight: Worker clusters invert the usual economics. Latency does not matter much, but cost per unit of work and graceful interruption dominate every design choice.
• Key Concepts: KEDA ScaledObject, scale-to-zero, spot interruption handling, PodDisruptionBudget, and idempotent, checkpointed work.
• When to Use: bursty or queue-driven workloads that tolerate restarts and where fixed node pools sit idle most of the day.
• Limitations: everything here assumes work is retryable. A non-idempotent worker on spot capacity will corrupt data, not save money.

Recipe

1. Make every worker idempotent and interruption-safe before touching autoscaling. This is the prerequisite, not step five.
2. Scale on queue depth with KEDA, not on CPU with the HPA.
3. Run workers on spot NodePools via Karpenter, with a small on-demand pool for control-plane-adjacent add-ons.
4. Handle interruption: terminationGracePeriodSeconds longer than your longest task, plus a preStop drain.
5. Guard the fleet with PodDisruptionBudgets so voluntary disruptions cannot empty a consumer group.

Working Example

KEDA scales the consumer from zero based on SQS depth. The HPA cannot do this - it has no queue awareness and will not scale to zero.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: transcode-worker
  namespace: media
spec:
  scaleTargetRef:
    name: transcode-worker
  minReplicaCount: 0
  maxReplicaCount: 400
  pollingInterval: 15
  cooldownPeriod: 300
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleDown:
          stabilizationWindowSeconds: 300
  triggers:
    - type: aws-sqs-queue
      metadata:
        queueURL: https://sqs.us-east-1.amazonaws.com/123456789012/transcode
        queueLength: "20"
        awsRegion: us-east-1
      authenticationRef:
        name: keda-aws-irsa

queueLength: "20" is the target backlog per pod. KEDA creates and owns an HPA underneath; never manage that HPA yourself.

The worker Deployment is where interruption safety lives.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: transcode-worker
  namespace: media
spec:
  replicas: 0
  selector:
    matchLabels: { app: transcode-worker }
  template:
    metadata:
      labels: { app: transcode-worker }
    spec:
      # longest single task is ~4 min; allow it to finish
      terminationGracePeriodSeconds: 300
      nodeSelector:
        karpenter.sh/capacity-type: spot
      securityContext:
        runAsNonRoot: true
        seccompProfile: { type: RuntimeDefault }
      containers:
        - name: worker
          image: 123456789012.dkr.ecr.us-east-1.amazonaws.com/transcode@sha256:abc123
          resources:
            requests: { cpu: "2", memory: 4Gi }
            limits: { memory: 6Gi }
          lifecycle:
            preStop:
              exec:
                # stop long-polling for new messages, finish the current one
                command: ["/bin/sh", "-c", "touch /tmp/drain && sleep 290"]
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities: { drop: ["ALL"] }

Karpenter provisions the spot capacity and consolidates it away when the queue drains.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: spot-workers
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64", "arm64"]
        - key: karpenter.k8s.aws/instance-family
          operator: In
          values: ["c6i", "c6a", "c7g", "m6i", "m6a"]
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      expireAfter: 168h
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 1m
  limits:
    cpu: "2000"

Deep Dive

Why breadth of instance types is the spot strategy

Spot interruption risk is per capacity pool, not per account. Allowing five instance families across two architectures gives Karpenter dozens of pools to choose from.

A NodePool pinned to one instance type on spot is the single most common cause of "spot is unreliable" complaints.

The interruption timeline

EC2 sends a spot interruption notice roughly two minutes before reclaim. Karpenter watches the interruption queue, cordons the node, and starts draining.

Two minutes is your hard ceiling. A terminationGracePeriodSeconds of 300 does not extend it - the node disappears regardless, so tasks longer than ~90 seconds need checkpointing, not just a longer grace period.

Scale-to-zero and cold start

minReplicaCount: 0 is the big saving and the big latency cost. From zero, a job waits for KEDA polling, then pod scheduling, then possibly a Karpenter node launch and image pull.

Budget 60 to 180 seconds. If that is unacceptable, set minReplicaCount: 1 and accept one warm pod per consumer.

Asymmetric scaling

Scale up fast, scale down slow. The cooldownPeriod and a 300-second stabilizationWindowSeconds stop a bursty queue from thrashing nodes up and down every poll.

Node churn costs money too - image pulls, warmup, and consolidation are not free.

Disruption budgets

Karpenter consolidation is a voluntary disruption, so it respects PDBs. Spot reclaim is involuntary and does not.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: transcode-worker
  namespace: media
spec:
  maxUnavailable: 25%
  selector:
    matchLabels: { app: transcode-worker }

Use maxUnavailable as a percentage, not minAvailable as a fixed number. With scale-to-zero, a minAvailable: 3 PDB blocks every drain once replicas drop below three.

Gotchas

A PDB that can never be satisfied blocks node upgrades forever. This is the classic worker-cluster outage: a stuck drain during a cluster upgrade, traced to a fixed minAvailable on a scale-to-zero deployment.

KEDA scales on backlog, not on progress. If workers crash-loop, the queue grows and KEDA scales up more crashing pods. Alert on queue depth and worker error rate.

Long grace periods do not survive spot reclaim. Checkpoint or shorten tasks. Two minutes is the real budget.

In-flight message visibility timeouts must exceed the grace period. Otherwise a message is redelivered while the original pod is still processing it.

Consolidation can undo your scale-up. With consolidateAfter: 1m and a spiky queue, Karpenter removes nodes you are about to need. Tune it up for spiky workloads.

Cluster Autoscaler and Karpenter must not both manage a pool. They will fight. Pick one.

Alternatives

HPA on CPU. Simple and built in, but blind to backlog and cannot reach zero. Fine for steady CPU-bound consumers.

Jobs and CronJobs per unit of work. Excellent isolation and natural retries, but pod-per-task overhead dominates for short tasks.

AWS Batch or serverless (Lambda, Fargate). Skip node management entirely. Better for spiky, short work; worse for GPU or long-running tasks with big images.

On-demand only. Two to three times the cost, far fewer failure modes. Legitimate for a small worker fleet where engineering time costs more than compute.

FAQs

Can KEDA and a plain HPA target the same Deployment? No. KEDA creates its own HPA. A second HPA on the same target causes both to fight.

What percentage of a worker fleet should be spot? Commonly 80 to 100% for retryable work, with on-demand reserved for KEDA, Karpenter, and CoreDNS themselves.

Do I still need requests and limits? Yes. Requests drive Karpenter's provisioning math directly, so bad requests produce wrong-sized nodes.

How do I keep KEDA itself off spot? Run add-ons on a small on-demand NodePool selected by nodeSelector, so the autoscaler is never reclaimed mid-scale.

What breaks first as this scales? Usually the queue's own limits or a downstream database connection pool, not Kubernetes. Load-test the dependency, not the cluster.

Related

• How to Read a Reference Architecture
• Reference: Multi-Tenant EKS Platform
• Reference: Regulated Environment
• Before/After: Compose to EKS Migration
• Case Studies Best Practices

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).