StatefulSets Best Practices

Running stateful workloads on Kubernetes is mostly about disciplined defaults around storage, identity, and rollout.

This list groups the practices that keep StatefulSets safe to operate, upgrade, and scale.

How to Use This List

Read the groups in order; each builds on the last from storage through operations.

Treat every bold item as a rule with a reason, and adapt the specifics to your database or clustered system.

Apply these before your first production StatefulSet, not after your first incident.

A - Storage and PVC Templates

Plan volumeClaimTemplates up front. Each pod gets its own PVC named <template>-<statefulset>-<ordinal>, and you cannot cleanly retrofit per-pod storage later, so design it before launch.
Pin a StorageClass explicitly. Name the storageClassName in the template rather than relying on a cluster default that may change or differ across environments.
Set a persistentVolumeClaimRetentionPolicy deliberately. Choose whenScaled and whenDeleted so you neither lose data nor silently accumulate orphaned volumes and cost.
Right-size storage requests. Most CSI drivers support volume expansion but not shrink, so start conservative and grow with allowVolumeExpansion when needed.
Use ReadWriteOnce per pod. Per-pod volumes should be exclusively owned; reserve ReadWriteMany for genuinely shared filesystems, not database data.

B - Identity and Networking

Always pair with a headless Service. Set clusterIP: None and match its name to serviceName, or pods never get stable per-member DNS.
Address peers by ordinal DNS. Use <pod>.<service>.<namespace>.svc.cluster.local for seeds and cluster membership instead of hardcoding IPs.
Add a second ClusterIP Service for clients. Keep peer discovery (headless) separate from load-balanced application traffic (ClusterIP).
Enable publishNotReadyAddresses only when bootstrapping needs it. It lets seed nodes resolve peers before readiness, at the cost of exposing not-ready addresses.
Never rely on pod IPs. IPs change on reschedule; only the ordinal name and PVC are stable, so build discovery on names.

C - Rollout and Updates

Keep OrderedReady for quorum systems. The default sequential rollout protects consensus databases; only switch to podManagementPolicy: Parallel for shared-nothing apps.
Canary with partitioned RollingUpdate. Set a partition to update the highest ordinals first, verify, then lower it step by step.
Gate readiness on real health. A readiness probe should reflect the app actually serving, because the next pod waits on it during ordered rollout.
Use a PodDisruptionBudget. Protect quorum during node drains and maintenance by bounding how many members can be unavailable at once.
Test upgrades on a copy first. Restore a snapshot into a scratch StatefulSet and rehearse the version bump before touching production.

D - Reliability and Scheduling

Spread pods across failure domains. Use topologySpreadConstraints or anti-affinity so members land on different nodes and zones, avoiding correlated loss.
Set requests and limits on every container. Stateful pods are long-lived; unbounded usage causes noisy-neighbor evictions that are expensive to recover from.
Add startup and liveness probes carefully. Give slow-starting databases generous initialDelaySeconds or a startupProbe so they are not killed mid-recovery.
Handle graceful shutdown. Set a realistic terminationGracePeriodSeconds and a preStop hook so a member leaves the cluster cleanly before the process stops.
Back up data independently of the volume. Snapshots protect against corruption and human error that replica volumes will faithfully copy.

E - Security and Supply Chain

Run as non-root. Set runAsNonRoot, a fixed runAsUser, and fsGroup so the pod owns its volume without root.
Pin images by digest and scan them. Reference immutable digests and gate on Trivy or Grype results; sign with cosign for provenance.
Apply the restricted Pod Security Standard. Stateful apps rarely need host access, so hold them to the restricted profile.
Default-deny network traffic. Use NetworkPolicies so only intended peers and clients can reach database ports.
Keep secrets out of images. Mount credentials from Secrets or an external secrets manager, never bake them into the image or manifest.

When You Are Done

You should have per-pod storage with an explicit class and retention policy, a headless Service for identity, and a client Service for traffic.

Your rollout should be ordered or intentionally parallel, guarded by probes, a PDB, and spread constraints.

Security should be non-root, digest-pinned, scanned, network-restricted, and running under the restricted profile.

FAQs

Do I always need a headless Service? Yes if pods must address each other by name. The StatefulSet references it through serviceName, and without it there are no per-pod DNS records.

What retention policy should I pick? For most databases, whenScaled: Retain and whenDeleted: Retain are safest, but set whenScaled: Delete if scale-in should reclaim storage automatically.

Is a StatefulSet enough for high availability? No. It provides identity and stable storage; replication, failover, and leader election are the application's responsibility.

How do I upgrade without downtime? Use partitioned RollingUpdate to canary high ordinals, keep readiness accurate, and protect quorum with a PodDisruptionBudget.

Can I change volumeClaimTemplates later? Not the storage size retroactively for existing PVCs beyond expansion, and template changes do not apply to already-created claims. Plan storage before launch.

Should StatefulSet pods run as root? No. Run as non-root with an explicit user and fsGroup, and hold them to the restricted Pod Security Standard.

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).

StatefulSets Best Practices

Running stateful workloads on Kubernetes is mostly about disciplined defaults around storage, identity, and rollout.

This list groups the practices that keep StatefulSets safe to operate, upgrade, and scale.

How to Use This List

Read the groups in order; each builds on the last from storage through operations.

Treat every bold item as a rule with a reason, and adapt the specifics to your database or clustered system.

Apply these before your first production StatefulSet, not after your first incident.

A - Storage and PVC Templates

Plan volumeClaimTemplates up front. Each pod gets its own PVC named <template>-<statefulset>-<ordinal>, and you cannot cleanly retrofit per-pod storage later, so design it before launch.
Pin a StorageClass explicitly. Name the storageClassName in the template rather than relying on a cluster default that may change or differ across environments.
Set a persistentVolumeClaimRetentionPolicy deliberately. Choose whenScaled and whenDeleted so you neither lose data nor silently accumulate orphaned volumes and cost.
Right-size storage requests. Most CSI drivers support volume expansion but not shrink, so start conservative and grow with allowVolumeExpansion when needed.
Use ReadWriteOnce per pod. Per-pod volumes should be exclusively owned; reserve ReadWriteMany for genuinely shared filesystems, not database data.

B - Identity and Networking

Always pair with a headless Service. Set clusterIP: None and match its name to serviceName, or pods never get stable per-member DNS.
Address peers by ordinal DNS. Use <pod>.<service>.<namespace>.svc.cluster.local for seeds and cluster membership instead of hardcoding IPs.
Add a second ClusterIP Service for clients. Keep peer discovery (headless) separate from load-balanced application traffic (ClusterIP).
Enable publishNotReadyAddresses only when bootstrapping needs it. It lets seed nodes resolve peers before readiness, at the cost of exposing not-ready addresses.
Never rely on pod IPs. IPs change on reschedule; only the ordinal name and PVC are stable, so build discovery on names.

C - Rollout and Updates

Keep OrderedReady for quorum systems. The default sequential rollout protects consensus databases; only switch to podManagementPolicy: Parallel for shared-nothing apps.
Canary with partitioned RollingUpdate. Set a partition to update the highest ordinals first, verify, then lower it step by step.
Gate readiness on real health. A readiness probe should reflect the app actually serving, because the next pod waits on it during ordered rollout.
Use a PodDisruptionBudget. Protect quorum during node drains and maintenance by bounding how many members can be unavailable at once.
Test upgrades on a copy first. Restore a snapshot into a scratch StatefulSet and rehearse the version bump before touching production.

D - Reliability and Scheduling

Spread pods across failure domains. Use topologySpreadConstraints or anti-affinity so members land on different nodes and zones, avoiding correlated loss.
Set requests and limits on every container. Stateful pods are long-lived; unbounded usage causes noisy-neighbor evictions that are expensive to recover from.
Add startup and liveness probes carefully. Give slow-starting databases generous initialDelaySeconds or a startupProbe so they are not killed mid-recovery.
Handle graceful shutdown. Set a realistic terminationGracePeriodSeconds and a preStop hook so a member leaves the cluster cleanly before the process stops.
Back up data independently of the volume. Snapshots protect against corruption and human error that replica volumes will faithfully copy.

E - Security and Supply Chain

Run as non-root. Set runAsNonRoot, a fixed runAsUser, and fsGroup so the pod owns its volume without root.
Pin images by digest and scan them. Reference immutable digests and gate on Trivy or Grype results; sign with cosign for provenance.
Apply the restricted Pod Security Standard. Stateful apps rarely need host access, so hold them to the restricted profile.
Default-deny network traffic. Use NetworkPolicies so only intended peers and clients can reach database ports.
Keep secrets out of images. Mount credentials from Secrets or an external secrets manager, never bake them into the image or manifest.

When You Are Done

You should have per-pod storage with an explicit class and retention policy, a headless Service for identity, and a client Service for traffic.

Your rollout should be ordered or intentionally parallel, guarded by probes, a PDB, and spread constraints.

Security should be non-root, digest-pinned, scanned, network-restricted, and running under the restricted profile.

FAQs

Do I always need a headless Service? Yes if pods must address each other by name. The StatefulSet references it through serviceName, and without it there are no per-pod DNS records.

What retention policy should I pick? For most databases, whenScaled: Retain and whenDeleted: Retain are safest, but set whenScaled: Delete if scale-in should reclaim storage automatically.

Is a StatefulSet enough for high availability? No. It provides identity and stable storage; replication, failover, and leader election are the application's responsibility.

How do I upgrade without downtime? Use partitioned RollingUpdate to canary high ordinals, keep readiness accurate, and protect quorum with a PodDisruptionBudget.

Should StatefulSet pods run as root? No. Run as non-root with an explicit user and fsGroup, and hold them to the restricted Pod Security Standard.

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).

StatefulSets Best Practices

Search Documentation

StatefulSets Best Practices