Lesson  in  Kubelings — Learn Kubernetes the Rustlings Way

The drain that never finishes

Node maintenance tonight. But kubectl drain has been "evicting" the same two pods for ten minutes — a PodDisruptionBudget demands more availability than the deployment has replicas. Learn the PDB math and fix it so maintenance and availability can both be true.

The situation

Kernel patch night. Runbook step 3: kubectl drain kubelings-worker. You run it and watch:

evicting pod kubelings/tickets-6d8f7b9c4-2xkpl
error when evicting pods/"tickets-6d8f7b9c4-2xkpl" -n "kubelings" (will retry after 5s):
Cannot evict pod as it would violate the pod's disruption budget.

…and again. And again. Ten minutes of the same two lines. The drain isn't broken — it's being refused, politely, forever.

A PodDisruptionBudget is a contract with the eviction API: "never let voluntary disruptions take availability below X." Drains, node upgrades, cluster-autoscaler scale-downs — all go through eviction and all must honor it.

Now the math on this one:

replicas:      2
minAvailable:  2
disruptionsAllowed = 2 - 2 = 0

Zero. Not "wait until it's safer" — structurally zero, at all times. Whoever wrote this PDB demanded 100% of replicas be up always, which outlaws maintenance itself.

   ┌──────────────┐  
   │kubectl drain │  
   │              │  
   └──────────────┘  
          │          
      evict pod      
          │          
          ▼          
    ┌─────────────┐  
    │eviction API │  
    │             │  
    └─────────────┘  
          │          
would breach, refused
          │          
          ▼          
  ┌─────────────────┐
  │PDB minAvailable │
  │                 │
  └─────────────────┘

Your task

Make maintenance possible without giving up protection:

  1. Look at what the PDB currently allows (kubectl get pdb shows the columns).
  2. Fix the contract so at least one disruption is allowed while the app stays protected — change the PDB's math, or give it more replicas to budget with. Keep a PDB either way.
  3. tickets must remain fully Available.
kubectl -n kubelings get pdb tickets-pdb
kubectl -n kubelings get deploy tickets
Hint

Two honest fixes:

# a) express the budget as tolerated disruption:
kubectl -n kubelings patch pdb tickets-pdb --type=merge \
  -p '{"spec":{"minAvailable":null,"maxUnavailable":1}}'

# b) or fund the budget with headroom:
kubectl -n kubelings scale deploy/tickets --replicas=3   # minAvailable:2 now allows 1

Then check: kubectl -n kubelings get pdb → ALLOWED DISRUPTIONS ≥ 1.

Solution

Root cause

minAvailable: 2 on a 2-replica deployment = disruptionsAllowed: 0. The eviction API did exactly its job: it refused every eviction, indefinitely. In real fleets this is how "one team's PDB" blocks cluster-wide node upgrades — the autoscaler and upgrade tooling retry forever against one immovable budget.

Fix

Either express tolerance:

kubectl -n kubelings patch pdb tickets-pdb --type=merge \
  -p '{"spec":{"minAvailable":null,"maxUnavailable":1}}'

or fund the budget:

kubectl -n kubelings scale deploy/tickets --replicas=3

Both end with ALLOWED DISRUPTIONS: 1 — drain proceeds one pod at a time, availability floor intact.

PDB math cheat sheet

replicasPDBallowed disruptions
2minAvailable: 20 — drains hang
3minAvailable: 21
2maxUnavailable: 11
1any PDB0 — singletons can't be protected and drainable

Prefer maxUnavailable — it keeps allowing disruption as you scale up, whereas minAvailable as an absolute number silently tightens when replicas shrink.

Prevention

  • CI-lint PDBs against replica counts: disruptionsAllowed == 0 at steady state is a bug, not a policy.
  • Involuntary disruptions (node crash, OOM) ignore PDBs entirely — a PDB is not HA; it only shapes voluntary churn.
  • Before maintenance windows: kubectl get pdb -A and look at the ALLOWED column. Zeroes = tonight's incident.
Previous lesson
Stuck at Init: the gate that never opens
Next lesson
QoS classes: who gets killed first
Start lesson