Jobs Best Practices

Batch workloads fail differently than services: they retry, accumulate, and run unattended on schedules. This list collects the practices that keep Jobs and CronJobs correct, cheap, and observable in production.

How to Use This List

Read the groups in order when you author a new Job, and treat each bold practice as a review checkbox on pull requests.

The groups move from correctness, to reliability, to cost and cleanup, to security, to operations. Apply every item that fits your workload rather than cherry-picking.

A - Correctness and Idempotency

• Make every Job idempotent. Retries, missed-then-recovered CronJob runs, and rare double-creation mean the same work can run more than once, so design it to be safe to repeat.
• Choose the right restartPolicy. Use Never for clean per-attempt Pods and logs, or OnFailure when you want fewer objects and in-place container restarts.
• Never model batch work as a Deployment. A one-shot task in a Deployment exits 0 and gets restarted forever; use a Job so a clean exit ends the work.
• Pick the correct completion mode. Use completionMode: Indexed when Pods own fixed shards, and NonIndexed when Pods pull interchangeable work.
• Validate exit codes deliberately. Ensure success paths exit 0 and real failures exit non-zero, because the Job controller trusts the exit code completely.

B - Reliability and Failure Handling

• Always cap retries with backoffLimit. The default is 6; set it explicitly so a broken image cannot retry endlessly and mask a deploy problem.
• Bound total runtime with activeDeadlineSeconds. This catches processes that hang instead of crashing, terminating the Job with DeadlineExceeded.
• Use podFailurePolicy for known-fatal exit codes. Fail fast with FailJob on unrecoverable codes, or Ignore infrastructure disruptions so they do not consume retry budget.
• Isolate bad shards in indexed Jobs. Set backoffLimitPerIndex and maxFailedIndexes so one poisoned partition does not fail the whole run.
• Set startingDeadlineSeconds on CronJobs. Without it, a long outage can trip the 100-missed-schedule limit and silently stop all future runs.

C - Concurrency and Scheduling

• Set concurrencyPolicy explicitly on CronJobs. The default Allow stacks overlapping runs; choose Forbid for backups and reports, or Replace when only the latest run matters.
• Always set timeZone on CronJobs. Use an IANA zone so "2 AM" means local 2 AM, and keep critical jobs off the daylight-saving transition hour.
• Tune parallelism to real capacity. Requesting more parallel Pods than the cluster can schedule just leaves Pods Pending; pair large fan-outs with node autoscaling.
• Balance shard sizes. Uneven partitions leave one Pod lagging while the rest idle, wasting the parallelism you paid for.

D - Cost, Cleanup, and Resources

• Set ttlSecondsAfterFinished on every Job. The TTL controller then deletes finished Jobs and their Pods, keeping etcd and namespaces clean.
• Bound CronJob history. Tune successfulJobsHistoryLimit and failedJobsHistoryLimit to keep a few runs for debugging without unbounded growth.
• Always set resource requests and limits. Requests let the scheduler pack Pods correctly, and a memory limit stops one batch Pod from starving co-tenants.
• Right-size for the batch, not the peak. Batch Pods run briefly, so provision requests for typical work rather than worst-case spikes to improve bin-packing.

E - Security and Supply Chain

• Run batch Pods as non-root. Set runAsNonRoot: true and a RuntimeDefault seccomp profile to align with the restricted Pod Security Standard.
• Pin and digest base images. Build with BuildKit from pinned or digest-referenced bases so a rebuild cannot silently pull a different image.
• Scan and sign images. Run Trivy or Grype in CI and sign with cosign, since batch images ship the same risks as service images.
• Scope credentials tightly. Give the Job's ServiceAccount only the RBAC and secrets it needs, because scheduled jobs are easy to over-privilege and forget.

F - Observability and Operations

• Emit structured logs and a final status line. Batch Pods have no readiness endpoint, so logs are your primary signal of what happened.
• Alert on Job failures and missed CronJobs. Watch the Failed condition and the CronJob's lastScheduleTime so a silently broken schedule surfaces quickly.
• Test runs with kubectl create job --from. Trigger a CronJob's work on demand to validate changes before trusting the schedule.
• Manage Jobs and CronJobs via GitOps. Keep manifests in Helm or Kustomize and reconcile with Argo CD so schedules are reviewable and reproducible.

When You Are Done

You should be able to point to explicit values for backoffLimit, activeDeadlineSeconds, and ttlSecondsAfterFinished on every Job, and timeZone, concurrencyPolicy, and startingDeadlineSeconds on every CronJob.

If any of those are defaulted by omission, treat it as a gap and set them before shipping.

FAQs

What is the single most forgotten field? ttlSecondsAfterFinished. Without it, finished Jobs pile up and slowly bloat the cluster.

How do I stop a Job retrying forever? Set backoffLimit and activeDeadlineSeconds. When either is exceeded, the Job is marked Failed and stops.

Why did my CronJob silently stop? It likely hit the 100-missed-schedule cliff after an outage; set startingDeadlineSeconds to prevent it.

Should batch Pods have probes? Skip readiness probes since nothing routes traffic to them, but a liveness probe or active deadline guards against hangs.

How do I keep one bad shard from failing everything? Use completionMode: Indexed with backoffLimitPerIndex and maxFailedIndexes to isolate failures.

Do batch images need the same security controls as services? Yes. Non-root, pinned bases, scanning, and signing all apply equally to Jobs.

Related

• Run-to-Completion vs Long-Running Work
• Jobs Basics
• CronJob Patterns
• Indexed & Parallel Jobs

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).