Run-to-Completion vs Long-Running Work
Summary
- Kubernetes workloads split into two families: long-running services that should never exit, and run-to-completion tasks that do a unit of work and stop.
- Insight: the controller you choose encodes an intent, and the scheduler, restart logic, and health model all follow from that intent.
- Key Concepts: Deployment/StatefulSet (long-running), Job/CronJob (run-to-completion), restartPolicy, completions, backoffLimit.
- When to Use: pick Job semantics for migrations, ETL, report generation, and one-off maintenance; pick Deployment semantics for APIs, web servers, and queue consumers that idle waiting for work.
- Limitations/Trade-offs: Jobs do not autoscale on request load and long-running controllers treat a clean exit(0) as a crash to restart.
- Related Topics: Jobs Basics, CronJob Patterns, Indexed and Parallel Jobs.
Foundations
Every containerized process eventually exits. The difference is whether exiting is success or failure.
A long-running workload is one where exit is always a problem. A web server that returns and stops serving traffic has failed, regardless of its exit code.
A run-to-completion workload is the opposite. The process is supposed to finish, and finishing cleanly is the definition of success.
Kubernetes models these two intents with different controllers. Deployments and StatefulSets keep a target number of Pods running forever. Jobs run Pods until a required number of successful completions is reached, then stop.
This is not a cosmetic distinction. The controller determines what happens on exit, how failures are counted, and when the workload is considered done.
Consider a Deployment. Its Pods use restartPolicy: Always, and the ReplicaSet behind it continuously reconciles toward the desired replica count.
If a Pod's container exits, the kubelet restarts it. If the whole Pod dies, the ReplicaSet creates a replacement. There is no concept of "done."
Now consider a Job. Its Pods use restartPolicy: OnFailure or Never, and the Job controller tracks successful completions against .spec.completions.
When enough Pods exit successfully, the Job is marked Complete and no more Pods are created. Finishing is the goal, not a failure to recover from.
Mechanics & Interactions
The mechanical hinge between the two families is restartPolicy combined with the owning controller.
Long-running controllers require restartPolicy: Always. The API server rejects a Deployment Pod template that sets anything else.
Jobs forbid Always. A Job Pod template must use OnFailure or Never, because "always restart" is incompatible with "run until success."
With restartPolicy: Never, a failed container leaves the Pod in a terminal Failed state, and the Job controller creates a brand-new Pod for the next attempt. This gives you a clean per-attempt Pod for logs and debugging.
With restartPolicy: OnFailure, the kubelet restarts the failed container in place, reusing the same Pod. Fewer Pod objects are created, but attempt history is compressed into container restart counts.
Retries in a Job are bounded by .spec.backoffLimit, which defaults to 6. Each failure increases an exponential backoff delay, capped at six minutes, before the next attempt.
Long-running workloads have no backoffLimit. The kubelet uses CrashLoopBackOff to slow restarts, but it never gives up.
Health checks also diverge. Long-running Pods rely on liveness and readiness probes to gate traffic and detect hangs.
Jobs typically skip readiness probes entirely, because nothing routes traffic to a batch Pod. A liveness probe can still catch a wedged batch process, but the more common guardrail is .spec.activeDeadlineSeconds, which caps total wall-clock runtime.
Completion counting is the other big difference. A Job with .spec.completions: 5 and .spec.parallelism: 2 runs up to two Pods at a time until five have succeeded.
A Deployment has .spec.replicas instead, which is a steady-state target rather than a finish line.
Advanced Considerations & Applications
The hardest failures come from using the wrong family for a workload.
A one-shot database migration wrapped in a Deployment is a classic trap. It runs, exits 0, and the ReplicaSet immediately restarts it, re-running the migration in a loop.
The reverse trap is a long-lived queue consumer modeled as a Job. It blocks forever waiting for messages, so the Job never completes and activeDeadlineSeconds eventually kills healthy work.
Queue consumers are genuinely ambiguous. If the consumer idles waiting for new work indefinitely, it is a long-running service and belongs in a Deployment.
If the consumer drains a finite, known work queue and then exits, it is batch work and belongs in a Job, often a parallel one. See Indexed and Parallel Jobs for that pattern.
Scheduled work is a third case. A nightly report is run-to-completion, but it needs a trigger, which is exactly what a CronJob provides by creating a Job on a schedule.
Cleanup matters more for Jobs than for services. Completed Jobs and their Pods linger by default, so set .spec.ttlSecondsAfterFinished to have the TTL controller garbage-collect them.
Resource requests apply to both families but mean different things. For a service, requests shape steady-state capacity and autoscaling; for a Job, requests mainly affect scheduling and how many parallel Pods can pack onto nodes.
On the runtime side, the same container image runs either way. Docker and BuildKit build the image for both, and containerd runs the resulting Pods on nodes under CRI regardless of whether the owner is a Deployment or a Job.
Common Misconceptions
"A Job is just a Pod that runs once." A Job is a controller that manages Pods, handling retries, parallelism, and completion counting. The bare Pod is only one attempt.
"Setting restartPolicy: Never turns a Deployment into a batch job." Deployments reject any restartPolicy other than Always. The controller, not the field alone, defines the semantics.
"exit(0) always means success everywhere." For a Job Pod it means success, but for a Deployment Pod a clean exit is treated as an outage and restarted.
"CronJobs are a separate kind of workload from Jobs." A CronJob is a scheduler that stamps out Job objects. Everything you know about Jobs applies to the Jobs it creates.
"Batch work should scale up under load like a service." Jobs scale by parallelism you set explicitly, not by request-driven autoscaling like a HorizontalPodAutoscaler on a Deployment.
FAQs
How do I know which controller to use? Ask whether a clean exit is success or failure. Success means Job or CronJob; failure means Deployment or StatefulSet.
Can a Job Pod use restartPolicy: Always? No. The API server rejects it, because a Job must be able to reach a terminal successful state.
What stops a failing Job from retrying forever? The backoffLimit caps retries and activeDeadlineSeconds caps total runtime. When either is exceeded the Job is marked Failed.
Do batch Pods need readiness probes? Usually not, since no Service routes traffic to them. A liveness probe or an active deadline is the better guardrail against a hung process.
Why does my one-off migration keep re-running? It is almost certainly a Deployment. Move it to a Job so a clean exit ends the work instead of triggering a restart.
Are StatefulSets long-running or batch? Long-running. They add stable identity and ordering on top of the same never-exit contract as Deployments.
Related
Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).