Kubernetes Workloads: StatefulSets

A StatefulSet gives each Pod a stable name, a stable network identity, and its own PersistentVolumeClaim. Pods are created in ordinal order and removed in reverse order. A Deployment treats all Pods as interchangeable - a StatefulSet does not. StatefulSets are used for databases where each node has a fixed role, for message queues where each consumer handles a fixed portion of work, and for any workload where each instance needs a stable identity.

Like DaemonSets, a StatefulSet stores rollout history in ControllerRevision objects rather than ReplicaSets - one object per unique pod template. The update order is different though: the controller replaces Pods in reverse ordinal order, db-3 first down to db-0, and waits for each Pod to be Ready before moving to the next. ControllerRevision is a standard Kubernetes API object - both DaemonSets and StatefulSets rely on it, and custom operators can use it too.

A headless Service is required for stable DNS identity. It gives each Pod a DNS entry in the form <pod>.<service>.<namespace>.svc.cluster.local. The StatefulSet references it by name in .spec.serviceName. Without it, pods fall back to an IP-based DNS name like 10-100-1-12.default.pod.cluster.local, which changes every time the pod is recreated.

Task 1 - Create a Headless Service

A headless Service has clusterIP: None. Kubernetes does not assign it a virtual IP - a DNS query for the Service name returns Pod IPs directly.

Steps:

Create a Service named db with .spec.clusterIP: None, selector app: db, port 80

Run kubectl get service db - the CLUSTER-IP column shows None. Run kubectl describe service db to see the Endpoints field, which is empty because no pods exist yet. The Service records clusterIP: None in its spec, which tells CoreDNS to return Pod IPs directly instead of a virtual IP.

Task 2 - Create a StatefulSet

Create a StatefulSet and watch Pods come up in strict ordinal order. The controller does not start db-1 until db-0 is Running and Ready.

StatefulSet - Kubernetes Docs

Steps:

Create a StatefulSet named db with .spec.serviceName: db, .spec.replicas: 4, selector app: db, one container named app using image busybox with command ["sleep", "3600"], terminationGracePeriodSeconds: 10 (optional - keeps the rollout observable without the default 30s wait per pod), a volume mount at /data named data, and a .spec.volumeClaimTemplates entry named data with 100Mi capacity and ReadWriteOnce access mode

.spec.serviceName is optional in the API - the StatefulSet runs without it. But without it, the controller sets no subdomain on the pods and CoreDNS registers no stable DNS records. Stable DNS identity requires this field to match the name of a headless Service in the same namespace.

Run kubectl get pods -l app=db - Pods are named db-0 through db-3, all Running. Run kubectl get pvc - four PVCs named data-db-0 through data-db-3, all Bound. Each PVC was provisioned automatically from the volumeClaimTemplates entry when its Pod was created. Run kubectl describe service db to see the Endpoints field, which now lists all four Pod IPs.

Task 3 - Query the Headless Service DNS

A regular Service has a ClusterIP - DNS returns that single virtual IP. A headless Service has no ClusterIP, so DNS returns individual Pod IPs directly, and each Pod also has its own record at <pod>.<service>.<namespace>.svc.cluster.local.

Stable Network Identity - Kubernetes Docs

Steps:

Exec into db-0 and run nslookup db-1.db.default.svc.cluster.local
Type the IP address from the output into the box below

nslookup db-1.db.default.svc.cluster.local returns a single A record for db-1. The name is stable - it resolves to whatever IP db-1 currently has, even after the Pod is recreated. Run kubectl get pods -l app=db -o wide to confirm the IP matches.

Task 4 - Scale Down and Back Up

When a StatefulSet scales down, Pods are removed in reverse ordinal order: db-3 first, then db-2. The PVCs are not deleted - they remain Bound until manually deleted. When the StatefulSet scales back up, the controller reuses those existing PVCs instead of creating new ones. db-2 reattaches to data-db-2 and db-3 reattaches to data-db-3.

Scaling a StatefulSet - Kubernetes Docs

Steps:

Scale db down to 2 replicas
Scale back up to 4 replicas

.spec.persistentVolumeClaimRetentionPolicy.whenScaled and .spec.persistentVolumeClaimRetentionPolicy.whenDeleted both default to Retain, which means PVCs are not deleted when pods scale down or when the StatefulSet itself is deleted. As long as the PVC exists, the PV and its data stay intact. If a PVC is deleted manually, the PV's reclaim policy takes over - dynamic provisioners default to Delete, which removes the underlying storage.

The StatefulSet uses the cluster's default StorageClass for PVC provisioning. To control the reclaim policy, either set a custom default StorageClass with the desired reclaimPolicy, or specify storageClassName directly in volumeClaimTemplates:

volumeClaimTemplates:
- metadata:
    name: data
  spec:
    storageClassName: my-storage-class
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 100Mi

After scaling down, run kubectl get pods -l app=db - only db-0 and db-1 remain. Run kubectl get pvc - all four PVCs are still Bound even though db-2 and db-3 no longer exist. After scaling back up, run kubectl get pod db-2 -o jsonpath='{.spec.volumes[*].persistentVolumeClaim.claimName}' - it shows data-db-2, the same PVC it had before. No new PVC was created.

Task 5 - Stable Identity

When a StatefulSet pod disappears - whether deleted, killed, or crashed - the controller recreates it with the same name, ordinal, and PVC.

Stable Storage - Kubernetes Docs

Steps:

Create the file /data/identity.txt in the volume mounted by db-1
Delete the db-1 Pod

Run kubectl get pods -l app=db - db-1 is back with a new AGE but the same name. Run kubectl get pod db-1 -o jsonpath='{.spec.volumes[*].persistentVolumeClaim.claimName}' - it still shows data-db-1. Run kubectl exec db-1 -- ls -l /data/identity.txt to confirm the file created before deletion is still there.

Task 6 - Roll an Image Update

The default update strategy for a StatefulSet is RollingUpdate. The controller replaces Pods in reverse ordinal order, waiting for each to be Ready before moving to the next. PVC attachments follow the Pod - data in the volume is not affected by an image update.

Update Strategies - Kubernetes Docs

Steps:

Update the db image to busybox:1.36

Each pod replacement waits for the previous pod to terminate before the new one starts. The default terminationGracePeriodSeconds is 30 seconds - with 4 replicas a full rollout would take 2+ minutes. This challenge sets it to 10 seconds so the rollout is fast enough to follow while still showing a visible Terminating state during each replacement.

Run kubectl rollout status sts/db to confirm the rollout is complete. Each pod template change creates a ControllerRevision object that stores the spec at that point. Run kubectl get controllerrevision -l app=db - two revisions now exist, one for the initial busybox template and one for busybox:1.36. Run kubectl get controllerrevision <name> -o yaml to inspect the .data field, which holds the patch applied at that revision.

Task 7 - Roll Back the Update

kubectl rollout undo finds the target ControllerRevision and restores the pod template from it. Pods are replaced in reverse ordinal order.

Rolling Back a StatefulSet - Kubernetes Docs

Steps:

Roll back db to the previous revision

Run kubectl rollout history sts/db - revision 1 is gone and revision 3 now points to the busybox template. The controller reused the existing ControllerRevision object and renumbered it from 1 to 3 rather than creating a new one. Run kubectl get controllerrevision -l app=db - two objects remain, numbered 2 and 3.

To target a specific revision rather than the previous one, run kubectl rollout undo sts/db --to-revision=<n>. The revision number maps directly to a ControllerRevision object.