Handling stateful services in containers

Why state is the hard part

Containers were designed around statelessness: a container can be killed and rescheduled anywhere, at any time, and that's a feature, not a bug. It's what makes auto-scaling and self-healing work. State breaks that assumption. A database container that gets rescheduled to a different node needs its data to follow it, or a way to reconnect to storage that stayed put, and it needs that to happen without corrupting anything mid-write.

None of this is Worldstream-specific: it's the same trade-off anyone running containers has to make, on any infrastructure. The two established patterns for dealing with it are covered below.

Pattern one: StatefulSets with persistent volumes

Kubernetes has a purpose-built object for this: the StatefulSet. Unlike a regular Deployment, a StatefulSet gives each pod a stable, unique network identity and a stable reference to its own persistent storage, so when a pod is rescheduled, it comes back with the same identity and reattaches to the same volume instead of starting from a blank slate.

The storage side of that is handled through PersistentVolumes and PersistentVolumeClaims: a PersistentVolumeClaim requests storage, and a PersistentVolume (backed by whatever storage class your cluster has configured) fulfils it and stays bound to that specific pod identity across reschedules. This is the standard, well-established way to run something like a small database or a message queue inside Kubernetes when you've decided it needs to live there.

It's still more operationally involved than a stateless Deployment. Things like resizing a volume, running an orderly failover, or restoring a specific replica after a crash need more care with a StatefulSet than with stateless pods, and it's worth going in with that expectation rather than discovering it during an incident.

Pattern two: don't put it in a container at all

The other legitimate answer is to not containerise the stateful part. Not entirely: containers optimise packaging and deployment; VMs provide stronger default isolation and OS flexibility. Many teams combine both, VMs for the boundaries that matter, containers for the parts that benefit from speed and repeatable deploys.

For a lot of teams, that means running the stateless application tier in containers (on Kubernetes or otherwise) while the database sits on a VPS or dedicated server, managed the way databases have always been managed: with proper backups, monitoring, and someone who understands its specific failure modes. That's not a compromise, it's often the simpler and more reliable choice, particularly if your team doesn't have deep day-to-day experience running stateful workloads on Kubernetes. See Do containers replace VMs? Choosing the right isolation model for that decision in more depth.

Whichever pattern you pick, plan the failure case

Containers sharing the host kernel is also a security and compliance consideration worth factoring in for stateful, often sensitive workloads: hardening (running rootless, using seccomp and AppArmor/SELinux profiles), image scanning and signing, and, where the compliance requirement calls for it, running the container inside a VM for stronger isolation.

Regardless of which pattern you land on, the thing that actually determines whether a stateful service survives a bad day is whether backup and restore has been tested, not just configured. A backup you've never restored from is a hope, not a plan. For the storage side of that, see the Storage category for Block Storage and Backup Storage guidance.