| Myth | Reality |
|---|---|
| Kubernetes was never meant for databases, so stateful workloads should stay on dedicated VMs. | StatefulSets, CSI drivers, and mature operators have made in-cluster databases a normal production pattern at companies running Postgres, Kafka, and Elasticsearch on Kubernetes today. |
| A Deployment with a PersistentVolumeClaim is functionally the same as a StatefulSet. | Deployments share a single ReplicaSet identity model; pods are interchangeable and volumes are not guaranteed to reattach to the same pod, which breaks quorum-based systems that depend on stable peer addresses. |
| Once you attach persistent storage, your data is automatically safe. | A PersistentVolume protects against pod rescheduling, not against accidental deletion, a bad storage class reclaim policy, or the absence of a tested backup and restore process. |
| Scaling a stateful app up or down is as simple as changing the replica count. | Ordinal-based scaling in a StatefulSet must respect quorum math, rebalancing time, and application-level rules, or you risk split-brain conditions and data loss during scale-in events. |
Why Stateful Workloads Are Different on Kubernetes
Kubernetes was originally designed around the assumption that pods are disposable. A Deployment can kill and recreate any pod at any time, reschedule it onto a different node, and give it a new IP address without anyone noticing, because the workload behind it does not care which specific instance answers a request. Stateful applications break that assumption in three ways: they need stable network identity so peers can find each other after a restart, they need the same durable storage volume reattached to the same logical instance, and they often need to start up or shut down in a specific order to preserve quorum or replication guarantees.
The StatefulSet controller exists specifically to address these three requirements. Instead of a randomly named pod like web-7d9f8c6b8-x2kpl, a StatefulSet produces pods with predictable, stable names such as postgres-0, postgres-1, and postgres-2. Each of those ordinals gets its own PersistentVolumeClaim created from a volumeClaimTemplate, and Kubernetes guarantees that when postgres-1 is rescheduled, it reattaches to the exact same underlying volume it had before, not a fresh one.
The Building Blocks: StatefulSets, PVCs, and Storage Classes
A StatefulSet on its own does not create storage. It relies on a chain of three Kubernetes primitives working together:
PersistentVolumeClaims and volumeClaimTemplates
When you define a volumeClaimTemplate inside a StatefulSet spec, Kubernetes stamps out one PersistentVolumeClaim per ordinal the first time the set scales up. Critically, scaling a StatefulSet back down does not delete those claims by default; the volumes persist so that scaling back up returns the same data to the same ordinal. This is a deliberate safety behavior, but it also means storage costs continue to accrue for scaled-in replicas unless you clean up claims manually or use the newer persistentVolumeClaimRetentionPolicy field to control that behavior explicitly.
StorageClasses and the Container Storage Interface
The actual disk behind a PVC is provisioned by a StorageClass, which delegates to a Container Storage Interface (CSI) driver specific to your infrastructure, whether that is Amazon EBS, Azure Disk, Google Persistent Disk, or an in-cluster system like Rook-Ceph, Longhorn, or Portworx. The StorageClass also sets the reclaim policy, which decides whether a released volume is deleted or retained, and the volume binding mode, which controls whether provisioning happens immediately or is delayed until a pod is actually scheduled so the volume can be placed in the correct availability zone.
Headless Services and Stable Network Identity
StatefulSets are paired with a headless Service (a Service with clusterIP: None) so that each pod gets its own DNS record following the pattern pod-name.service-name.namespace.svc.cluster.local, with the actual pod name, service name, and namespace substituted in. This is what lets a Postgres replica find its primary by a fixed hostname like postgres-0.postgres-headless rather than an ephemeral IP that changes on every reschedule.
Figure 1: StatefulSet Storage Binding Chain
Each ordinal pod (app-0, app-1, app-2) binds to its own PersistentVolumeClaim, which a StorageClass provisions through a CSI driver into a specific PersistentVolume. On rescheduling, the ordinal reattaches to the same PVC, not a new one, preserving data continuity.
Ordered Deployment and PodManagementPolicy
By default, a StatefulSet uses OrderedReady pod management: pods are created and terminated one at a time, in strict ordinal sequence, and each pod must report Ready before the next one starts. This matters enormously for systems like etcd or ZooKeeper-based clusters where a new member joining before the previous one has stabilized can corrupt quorum. For workloads that do not need this constraint, such as stateless-ish sharded caches where startup order is irrelevant, setting podManagementPolicy: Parallel lets all pods start simultaneously, which speeds up scaling considerably.
Rolling updates follow the same ordinal discipline. A StatefulSet updates pods from the highest ordinal down to the lowest by default, and you can pin an updateStrategy.rollingUpdate.partition value to canary a change on only the newest replicas before rolling it out cluster-wide, a technique commonly used when validating a new database minor version against read replicas first.
| Primitive | Role in Stateful Workloads | Common Pitfall |
|---|---|---|
| StatefulSet | Stable pod identity and ordinal scaling | Assuming it behaves like a Deployment for rollouts |
| PersistentVolumeClaim | Per-ordinal durable storage binding | Forgetting claims survive scale-down by default |
| StorageClass / CSI driver | Provisions and manages the underlying disk | Wrong reclaim policy deletes data on cleanup |
| Headless Service | Stable DNS identity per pod | Omitting it breaks peer discovery after restarts |
| PodDisruptionBudget | Limits concurrent voluntary evictions | Set too loosely, allowing quorum loss during upgrades |
Operators: Encoding Database Expertise as Code
Raw StatefulSets handle identity and storage, but they know nothing about database-specific operations like promoting a replica to primary, taking a consistent backup, or running a schema migration safely. That operational knowledge is captured by Kubernetes Operators, which extend the API with Custom Resource Definitions and a controller loop that reconciles desired state. The Zalando Postgres Operator and Crunchy Data’s PGO manage failover, connection pooling, and point-in-time recovery for Postgres clusters. Strimzi does the equivalent for Apache Kafka, managing broker rebalancing and topic configuration as Kubernetes-native resources. Elastic’s ECK operator handles shard allocation awareness for Elasticsearch clusters running on Kubernetes nodes.
Choosing between a hand-rolled StatefulSet and an operator is usually a maturity decision. Early-stage teams sometimes hand-roll a StatefulSet plus a headless Service and manage failover manually or through a sidecar. Once that same system is running in production with real uptime requirements, an operator that automates failover detection and safe reconfiguration becomes worth the added dependency, because a manual failover run at 3 a.m. during an incident is exactly where operators earn their keep.
Backup, Restore, and Disaster Recovery
Durable storage is not a backup strategy. A PersistentVolume protects against pod rescheduling and node failure within the boundaries of your storage system’s own replication, but it does nothing against an accidental DROP TABLE, a bad migration, or a ransomware event that touches every replica identically. Two mechanisms matter here:
VolumeSnapshots
The Kubernetes VolumeSnapshot API lets a CSI-compatible storage backend take a point-in-time, often crash-consistent, snapshot of a PVC that can be restored into a new claim. This is fast and storage-native, but it is only as good as your snapshot’s consistency guarantees relative to the application’s write pattern.
Application-Aware Backup Tools
Velero is the most widely deployed open-source tool for backing up Kubernetes resources and their associated persistent volumes together, and it supports hooks that let a database flush buffers or enter backup mode before the snapshot is taken, producing an application-consistent rather than merely crash-consistent backup. For most production stateful workloads, the sane baseline is scheduled Velero backups to a separate object store bucket, combined with periodic restore drills so the recovery path is actually exercised rather than assumed to work.
Figure 2: Backup Consistency Spectrum
Crash-consistent storage snapshots are fast but risk capturing a database mid-write. Application-aware backups that flush or checkpoint before snapshotting cost more time but guarantee a restorable, consistent state.
Common Mistakes When Running Stateful Workloads
Pitfall
Teams routinely set a StorageClass reclaim policy to Delete in every environment, including production, because that is the default on most managed Kubernetes distributions. When a PVC is accidentally removed during a namespace cleanup, the underlying cloud disk is deleted along with it, with no recycle bin. Setting the reclaim policy to Retain for anything holding production data, and cleaning up orphaned volumes deliberately, is a cheap insurance policy.
Pitfall
Another frequent mistake is omitting a PodDisruptionBudget on a quorum-based StatefulSet. Without one, a routine node drain during a cluster upgrade can evict two members of a three-node etcd or ZooKeeper ensemble simultaneously, taking the whole system below quorum. A PDB with maxUnavailable: 1 forces the scheduler to respect quorum math during any voluntary disruption.
Pitfall
Teams also under-provision anti-affinity rules, letting all three replicas of a database land on the same underlying node or availability zone. When that node fails, the “highly available” cluster goes down entirely. Pod anti-affinity rules, or the gentler topologySpreadConstraints, should be a non-negotiable part of any stateful workload’s spec.
Expert Practice: A Production-Grade Postgres Rollout
Worked Example
A platform team migrating a customer-facing Postgres cluster onto Kubernetes structured the rollout in four stages. First, they deployed the Zalando Postgres Operator into a staging namespace and validated failover by manually killing the primary pod, confirming a replica was promoted within the operator’s expected window. Second, they set the StorageClass reclaim policy to Retain, added a PodDisruptionBudget capping unavailability at one pod, and applied topologySpreadConstraints across three availability zones. Third, they scheduled nightly Velero backups with a pre-backup hook that ran pg_start_backup, storing artifacts in a separate region from the cluster itself. Fourth, before declaring production readiness, they ran a full restore drill into an isolated namespace and timed it, discovering the initial restore took twice as long as their recovery time objective allowed, which led them to add a warm standby replica specifically to shorten failover time. Only after that drill passed did the cluster take production traffic.
Scaling Considerations for Quorum-Based Systems
Scaling a stateless web tier is trivial: add replicas, let the load balancer distribute traffic. Scaling a StatefulSet running a quorum-based system like etcd, ZooKeeper, or a consensus-replicated database is not, because adding or removing a member changes the quorum size and can briefly reduce fault tolerance mid-transition. The safe pattern is to scale one ordinal at a time, waiting for the new member to fully sync and rejoin quorum before adding the next, and to always keep the total member count at an odd number to avoid split-vote scenarios. Scale-in is the more dangerous direction: removing a healthy voting member from an already-degraded cluster can push it below quorum instantly, so scale-in operations should always check current cluster health first, not just current replica count.
This ordering discipline is one reason many teams pair Kubernetes-native scaling with an operator rather than editing the replicas field directly; the operator can refuse or sequence a scale-down request based on live cluster health rather than blindly executing it.
Networking and Multi-Cluster Considerations
Stateful workloads rarely live in isolation from the rest of an organization’s infrastructure strategy. Teams running a multi-cloud architecture need to decide whether a stateful cluster spans clouds (introducing significant cross-cloud latency into replication) or stays single-cloud with asynchronous cross-region replication for disaster recovery instead. Regulated industries evaluating a hybrid cloud approach often keep the primary stateful database on-premises for data residency reasons while running stateless application tiers in the public cloud, which requires careful attention to the network path latency between the two. Teams choosing between orchestration and simpler compute models should also weigh the tradeoffs covered in our comparison of serverless, containers, and VMs, since stateful workloads are one of the strongest arguments for sticking with container orchestration rather than serverless, where storage and connection lifecycle assumptions rarely fit.
Event-driven systems add another layer: a Kafka cluster running as a StatefulSet is itself a piece of stateful infrastructure underpinning an event-driven architecture, which means its own durability guarantees become a dependency for every downstream consumer. Teams decomposing a large application, as described in our guide to moving from a monolith to a modular monolith, frequently discover that the shared database is the hardest piece to split apart precisely because of the stateful guarantees discussed here. And as AI workloads increasingly share the same clusters as traditional stateful services, understanding how Kubernetes schedules GPU workloads for AI is a natural next step for platform teams supporting both kinds of demanding workloads.
Glossary
- StatefulSet
- A Kubernetes controller that manages pods with stable, unique network identities and stable storage, created and scaled in a defined order.
- PersistentVolumeClaim (PVC)
- A request for storage by a user, bound to a matching PersistentVolume that satisfies its size and access requirements.
- CSI driver
- A Container Storage Interface plugin that lets Kubernetes provision, attach, and manage storage from a specific backend such as a cloud disk service or distributed storage system.
- Headless Service
- A Kubernetes Service with no cluster IP that returns individual pod DNS records instead of load-balancing traffic across them.
- PodDisruptionBudget
- A policy that limits how many pods in a set can be voluntarily disrupted at once, protecting quorum during maintenance operations.
- Operator
- A controller built on Custom Resource Definitions that automates application-specific operational tasks such as failover, backup, and scaling.
Key Takeaways
- StatefulSets provide stable identity and ordered scaling that Deployments cannot guarantee for quorum-based systems.
- PersistentVolumeClaims created from volumeClaimTemplates reattach to the same ordinal after rescheduling, preserving data continuity.
- Storage durability is not a backup strategy; snapshots and application-aware tools like Velero are still required.
- PodDisruptionBudgets and topology spread constraints prevent routine maintenance from accidentally breaking quorum.
- Operators encode database-specific operational knowledge that raw StatefulSets cannot provide on their own.
- Scale-in operations on quorum-based systems should check live cluster health, not just the replica count.
- Restore drills, not backup schedules alone, are what actually validate a disaster recovery plan.
FAQs
What is the difference between a StatefulSet and a Deployment?
A Deployment treats pods as interchangeable and gives no guarantee about which volume or network identity a rescheduled pod receives. A StatefulSet assigns each pod a fixed ordinal name, its own persistent volume claim, and enforces ordered startup and shutdown, which matters for databases and quorum-based clustered systems.
Do PersistentVolumeClaims get deleted automatically when I scale a StatefulSet down?
By default, no. Scaling down leaves the claims in place so that scaling back up restores the same data to the same ordinal. Kubernetes added a configurable persistentVolumeClaimRetentionPolicy field so teams can opt into automatic deletion on scale-down or deletion of the whole set when that behavior is actually desired.
Can I run a production database directly on Kubernetes without an operator?
Yes, but you take on manual responsibility for failover detection, backup orchestration, and safe upgrades that an operator would otherwise automate. Small or low-criticality databases sometimes run this way successfully, but any workload with real uptime requirements benefits from an operator that codifies those operational procedures.
What storage backend should I use for stateful workloads on Kubernetes?
It depends on your environment. Managed cloud disks through the native CSI driver are the simplest choice on a single public cloud. In-cluster systems like Rook-Ceph, Longhorn, or Portworx suit on-premises or hybrid deployments needing storage that lives inside the cluster itself. The right answer depends on latency, replication, and operational overhead tolerance.
How do I prevent a node failure from taking down my entire stateful cluster?
Use pod anti-affinity rules or topology spread constraints so replicas never land on the same node or availability zone, combine that with a PodDisruptionBudget to cap concurrent voluntary evictions, and confirm your storage backend replicates data across the same failure domains your pods are spread across.
What is the safest way to scale down a quorum-based StatefulSet?
Check the cluster’s current health before removing any member, remove one ordinal at a time rather than several simultaneously, and keep the remaining member count odd to avoid split-vote scenarios. Many production teams delegate this logic to an operator instead of editing the replica count directly.
Are VolumeSnapshots enough for disaster recovery on their own?
Not by themselves. Snapshots are often crash-consistent rather than application-consistent, meaning they can capture a database mid-write. Pairing snapshots with application-aware backup tools that support pre-backup hooks, and running periodic restore drills, is necessary to have a disaster recovery plan you can actually trust.
Should Kafka run as a StatefulSet on Kubernetes?
Yes, and it is one of the more common stateful workloads run this way, typically through the Strimzi operator, which manages broker identity, storage, and rebalancing as Kubernetes-native resources. Running Kafka this way brings the same identity and storage guarantees needed by any quorum-sensitive distributed system.
- Kubernetes Documentation, “StatefulSets,” kubernetes.io/docs/concepts/workloads/controllers/statefulset/
- Kubernetes Documentation, “Persistent Volumes,” kubernetes.io/docs/concepts/storage/persistent-volumes/
- Kubernetes Documentation, “Volume Snapshots,” kubernetes.io/docs/concepts/storage/volume-snapshots/
- Container Storage Interface Project, “CSI Specification,” kubernetes-csi.github.io/docs/
- CNCF, “Velero Project Documentation,” velero.io
- Strimzi Project, “Apache Kafka on Kubernetes Documentation,” strimzi.io
- Zalando, “Postgres Operator for Kubernetes,” github.com/zalando/postgres-operator
