PodDisruptionBudget and node draining

Sooner or later you have to touch a node: update the kernel, upgrade the Kubernetes version, change the machine type. That means emptying it of Pods, and that is where the question appears that separates a quiet maintenance from a call at three in the morning: how many replicas of my application can disappear at once without the service going down?

Kubernetes distinguishes two kinds of disruption:

Involuntary: a node goes down, runs out of memory, the rack catches fire. Nobody asks for them and nobody can negotiate them.
Voluntary: you drain a node, the autoscaler retires it, an operator replaces Pods. These can be negotiated, and the PodDisruptionBudget is the contract of that negotiation.

Work from the dev-machine tab.

Step 1: A spread-out application and its budget

Create web.yaml, with the Deployment and the PDB together:

cat << 'EOF' > web.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: ScheduleAnyway
        labelSelector:
          matchLabels:
            app: web
      containers:
      - name: nginx
        image: ghcr.io/iximiuz/labs/nginx:alpine
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: 20m
            memory: 32Mi
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web-pdb
spec:
  minAvailable: 3
  selector:
    matchLabels:
      app: web
EOF

The YAML, explained in questions and answers

What do the topologySpreadConstraints do?

They spread the Pods evenly along one dimension of the topology. topologyKey: kubernetes.io/hostname means "spread by node"; maxSkew: 1 means "between the most loaded node and the least loaded one there must not be more than one Pod of difference". With 3 replicas and the 2 nodes of this playground, the most balanced spread possible is 2-1, and that is exactly what you will see: maxSkew: 1 does not demand one Pod per node, it demands that the difference not exceed one. With 3 nodes you would get one per node, which is the case the book's map draws. Changing the key to topology.kubernetes.io/zone spreads by availability zone, which is how you survive a data center going down.

And whenUnsatisfiable?

What to do if the constraint cannot be met. DoNotSchedule turns it into a hard requirement (the Pod stays in Pending rather than unbalance the spread). ScheduleAnyway turns it into a preference: the scheduler tries to spread, but if it cannot, it places the Pod anyway. Here we use the soft version on purpose, because in a while we are going to cordon a node and we don't want the third replica left with nowhere to go.

Why this instead of a pod anti-affinity?

Because anti-affinity is binary (either there is a Pod on the node, or there isn't) and topologySpreadConstraints lets you say "spread, but allow some imbalance". It is the modern tool for what used to be done with the anti-affinity hammer.

What exactly does minAvailable: 3 declare?

That at all times there must be at least 3 Pods available among those matching the selector. Since the Deployment has exactly 3 replicas, we have just signed a contract that is impossible to honor if anyone wants to retire even one.

Is it the same as maxUnavailable?

They are two sides of the same coin, and you declare one or the other. minAvailable: 2 and maxUnavailable: 1 mean the same thing with 3 replicas, but behave differently if someone scales. The most robust form is the percentage (minAvailable: 50%), because it survives changes in the Deployment's size.

Who does a PDB really protect against?

Only against voluntary disruptions. A PDB does not stop a node from dying suddenly or a Pod from being evicted for lack of memory. It is not high availability: it is a handbrake for the operations that are done on purpose.

Apply it and look at the two things that matter:

kubectl apply -f web.yaml

kubectl get pods -l app=web -o wide

kubectl get pdb web-pdb

Three Pods spread across the two nodes: two on one and one on the other, which is the most balanced spread possible here. And in the PDB, notice the ALLOWED DISRUPTIONS column: it is 0. You have just declared, without meaning to, that this service allows no maintenance at all.

Step 2: The drain that gets blocked

Pick a node that has web Pods (any of the workers will do) and drain it, as you would before shutting it down:

kubectl get pods -l app=web -o wide
kubectl drain node-02 --ignore-daemonsets --delete-emptydir-data --timeout=30s

What exactly does drain do? Two things in a row: first it cordons the node (cordon, marks it as unschedulable, so no new Pods arrive) and then it evicts (evict) the Pods it already has. And that eviction is not a delete: it is an API request that goes through the PodDisruptionBudget filter.

That is why the command fails, with a message you already know how to read:

Cannot evict pod as it would violate the pod's disruption budget.

The node is left cordoned (check it with kubectl get nodes, it will say SchedulingDisabled), but the Pod is still there. The budget has done its job: it chose to block your maintenance rather than risk the service. A miscalculated PDB does not break the application, it breaks your operations, and that is why it is a failure that always shows up at the worst moment.

The flags, while you're at it:

--ignore-daemonsets: without it, drain refuses to start. A DaemonSet's Pods cannot really be evicted, because the controller would recreate them on the spot on that same node (remember the lesson: they are tied to their node).
--delete-emptydir-data: confirms that you accept losing the data in the emptyDir volumes of the evicted Pods. Kubernetes makes you say it out loud.

Step 3: A budget that lets you work

Fix the contract. With 3 replicas, requiring 2 available leaves room to retire one:

kubectl patch pdb web-pdb --type='merge' -p='{"spec":{"minAvailable":2}}'
kubectl get pdb web-pdb

The ALLOWED DISRUPTIONS column is now 1. That number is the one to look at before any maintenance, and the one to watch in production: a PDB with 0 allowed disruptions permanently is a time bomb for the day the cluster needs to move that Pod.

Repeat the drain:

kubectl drain node-02 --ignore-daemonsets --delete-emptydir-data

kubectl get pods -l app=web -o wide

kubectl get deployment web

Now it works. The Pod is evicted, the ReplicaSet creates the replacement on another node (that is why the spread was a preference and not a requirement), and the Deployment goes back to 3/3. Service intact, node empty and ready for maintenance.

Step 4: Give the node back to the cluster

The step everyone forgets, and the one that leaves clusters at half capacity for weeks:

kubectl uncordon node-02
kubectl get nodes

Note

💡 A PDB can block a drain forever if the Pods it protects never become Ready (for example, a replica in CrashLoopBackOff that prevents reaching the minAvailable). For that dead end there is unhealthyPodEvictionPolicy: AlwaysAllow, which allows evicting Pods that are not even ready. And there is also the dangerous shortcut, kubectl drain --disable-eviction, which ignores PDBs entirely: use it only when you know exactly what you are throwing away.

Summary

drain = cordon (no new Pods come in) + evict (the ones already there leave), and the eviction respects the PodDisruptionBudgets.
A PDB only governs voluntary disruptions. It is no substitute for having replicas.
minAvailable equal to the number of replicas means zero possible maintenances. Always look at the ALLOWED DISRUPTIONS column.
topologySpreadConstraints spreads replicas by node or by zone; with ScheduleAnyway it is a preference, with DoNotSchedule a requirement.
After draining, uncordon. Always.

Course progress