Storage Architecture Best Practices

How to Use This List

These are the storage decisions that matter most when running stateful workloads on Kubernetes.

Read them as defaults, not laws; deviate only when you can name the constraint that forces it.

The groups move from architecture, to StorageClass design, to operations, to security, to cost.

A - Architecture and Data Placement

Prefer managed data services for production databases. RDS, Cloud SQL, and Azure Database absorb failover, patching, and PITR; run systems of record there unless you have an operator and a real portability reason.
Keep the app stateless, push state to the edge of the cluster. Stateless Deployments scale and reschedule freely; concentrate durability in a few well-run stateful components or external services.
Match the driver to the access pattern, not the brand. Use single-writer block storage (EBS, PD, Azure Disk) for databases and shared file storage (EFS, Filestore, Azure Files) only when you truly need ReadWriteMany.
Use StatefulSets with volumeClaimTemplates for clustered data. They give stable identity and per-replica disks so each replica reclaims its own PV after rescheduling.
Respect zonal topology. Zonal block volumes pin Pods to a zone, so spread replicas across zones and let the scheduler honor CSI topology.

B - StorageClass Design

Expose a small, named menu of StorageClasses. Offer tiers like fast-ssd, standard, and archive rather than letting every team hand-tune volume parameters.
Set volumeBindingMode to WaitForFirstConsumer on zonal classes. This provisions the disk in the Pod's zone and avoids the Pending deadlock where a volume lands where no Pod can schedule.
Enable allowVolumeExpansion by default. Online growth is supported by most CSI drivers, and shrinking never is, so plan to only ever expand.
Encrypt at rest at the StorageClass level. Put encrypted: "true" (or the cloud equivalent) in class parameters so every volume is encrypted without per-PVC effort.
Choose reclaim policy deliberately. Delete is fine for reproducible data; use Retain for anything whose accidental deletion would hurt.

C - Backup and Recovery

Back up both objects and volume data. Ensure snapshots are enabled (Velero EnableCSI or snapshotVolumes: true) so restores do not come back with empty PVCs.
Run restore drills on a schedule. A snapshot is only a backup once you have restored it into a scratch namespace and verified the data.
Store backups in a separate account and region. Isolate the blast radius so one account compromise or regional outage cannot destroy source and backups together.
Prefer application-consistent backups for databases. Use backup hooks to quiesce, or take native dumps and WAL/PITR alongside crash-consistent volume snapshots.
Record your measured RTO and RPO. Turn recovery objectives into numbers proven by drills, not aspirations in a doc.

D - Security and Isolation

Run storage workloads as non-root. Set runAsNonRoot, an appropriate fsGroup, and target the restricted Pod Security Standard for stateful Pods.
Scope PVC and StorageClass access with RBAC. Limit who can create classes and claim expensive tiers so cost and blast radius stay controlled.
Never route Kubernetes storage through the Docker Engine. On nodes the CSI node plugin plus containerd handle mounts; dockershim is gone and Docker is a build and local-dev tool only.
Isolate stateful namespaces with NetworkPolicies. Default-deny, then allow only the app and backup traffic that a database actually needs.

E - Operations and Cost

Clean up orphaned PVCs after scale-down. StatefulSet scale-down keeps PVCs by design, so reclaim disks you no longer need to stop silent cost growth.
Right-size requests and limits on stateful Pods. Keep memory requests and limits equal for databases to get Guaranteed QoS and avoid OOM-driven eviction.
Monitor volume capacity and IOPS. Alert before a PV fills, since a full database disk is an outage, and watch for throttling on shared file tiers.
Test node and zone failure. Verify that a lost node reattaches its volumes elsewhere and that the operator promotes a survivor when a zone drops.

When You Are Done

You should be able to name the StorageClass tier and reclaim policy behind every production PVC.

You should have a scheduled backup with a rehearsed restore and a recorded RTO.

If any production database is a bare StatefulSet with no operator and no tested restore, fix that before anything else on this list.

FAQs

What is the single most important practice here? Prefer managed data services for production systems of record; it removes the hardest operational risk.

Do these apply to caches too? Partly; caches tolerate in-cluster storage and looser backups because the data is rebuildable.

Why WaitForFirstConsumer everywhere zonal? It keeps the disk and Pod in the same zone, preventing the most common Pending scheduling deadlock.

Is Retain safer than Delete? For irreplaceable data yes, at the cost of manual cleanup and orphaned-disk tracking.

How often should restore drills run? Frequently enough to trust the RTO; monthly is a common cadence and quarterly a floor.

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).

Storage Architecture Best Practices

How to Use This List

These are the storage decisions that matter most when running stateful workloads on Kubernetes.

Read them as defaults, not laws; deviate only when you can name the constraint that forces it.

The groups move from architecture, to StorageClass design, to operations, to security, to cost.

A - Architecture and Data Placement

Prefer managed data services for production databases. RDS, Cloud SQL, and Azure Database absorb failover, patching, and PITR; run systems of record there unless you have an operator and a real portability reason.
Keep the app stateless, push state to the edge of the cluster. Stateless Deployments scale and reschedule freely; concentrate durability in a few well-run stateful components or external services.
Match the driver to the access pattern, not the brand. Use single-writer block storage (EBS, PD, Azure Disk) for databases and shared file storage (EFS, Filestore, Azure Files) only when you truly need ReadWriteMany.
Use StatefulSets with volumeClaimTemplates for clustered data. They give stable identity and per-replica disks so each replica reclaims its own PV after rescheduling.
Respect zonal topology. Zonal block volumes pin Pods to a zone, so spread replicas across zones and let the scheduler honor CSI topology.

B - StorageClass Design

Expose a small, named menu of StorageClasses. Offer tiers like fast-ssd, standard, and archive rather than letting every team hand-tune volume parameters.
Set volumeBindingMode to WaitForFirstConsumer on zonal classes. This provisions the disk in the Pod's zone and avoids the Pending deadlock where a volume lands where no Pod can schedule.
Enable allowVolumeExpansion by default. Online growth is supported by most CSI drivers, and shrinking never is, so plan to only ever expand.
Encrypt at rest at the StorageClass level. Put encrypted: "true" (or the cloud equivalent) in class parameters so every volume is encrypted without per-PVC effort.
Choose reclaim policy deliberately. Delete is fine for reproducible data; use Retain for anything whose accidental deletion would hurt.

C - Backup and Recovery

Back up both objects and volume data. Ensure snapshots are enabled (Velero EnableCSI or snapshotVolumes: true) so restores do not come back with empty PVCs.
Run restore drills on a schedule. A snapshot is only a backup once you have restored it into a scratch namespace and verified the data.
Store backups in a separate account and region. Isolate the blast radius so one account compromise or regional outage cannot destroy source and backups together.
Prefer application-consistent backups for databases. Use backup hooks to quiesce, or take native dumps and WAL/PITR alongside crash-consistent volume snapshots.
Record your measured RTO and RPO. Turn recovery objectives into numbers proven by drills, not aspirations in a doc.

D - Security and Isolation

Run storage workloads as non-root. Set runAsNonRoot, an appropriate fsGroup, and target the restricted Pod Security Standard for stateful Pods.
Scope PVC and StorageClass access with RBAC. Limit who can create classes and claim expensive tiers so cost and blast radius stay controlled.
Never route Kubernetes storage through the Docker Engine. On nodes the CSI node plugin plus containerd handle mounts; dockershim is gone and Docker is a build and local-dev tool only.
Isolate stateful namespaces with NetworkPolicies. Default-deny, then allow only the app and backup traffic that a database actually needs.

E - Operations and Cost

Clean up orphaned PVCs after scale-down. StatefulSet scale-down keeps PVCs by design, so reclaim disks you no longer need to stop silent cost growth.
Right-size requests and limits on stateful Pods. Keep memory requests and limits equal for databases to get Guaranteed QoS and avoid OOM-driven eviction.
Monitor volume capacity and IOPS. Alert before a PV fills, since a full database disk is an outage, and watch for throttling on shared file tiers.
Test node and zone failure. Verify that a lost node reattaches its volumes elsewhere and that the operator promotes a survivor when a zone drops.

When You Are Done

You should be able to name the StorageClass tier and reclaim policy behind every production PVC.

You should have a scheduled backup with a rehearsed restore and a recorded RTO.

If any production database is a bare StatefulSet with no operator and no tested restore, fix that before anything else on this list.

FAQs

What is the single most important practice here? Prefer managed data services for production systems of record; it removes the hardest operational risk.

Do these apply to caches too? Partly; caches tolerate in-cluster storage and looser backups because the data is rebuildable.

Why WaitForFirstConsumer everywhere zonal? It keeps the disk and Pod in the same zone, preventing the most common Pending scheduling deadlock.

Is Retain safer than Delete? For irreplaceable data yes, at the cost of manual cleanup and orphaned-disk tracking.

How often should restore drills run? Frequently enough to trust the RTO; monthly is a common cadence and quarterly a floor.

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).

Storage Architecture Best Practices

Buscar en la documentación

Storage Architecture Best Practices