Recover Pods Stuck in Pending II
Scenario
The staging namespace contains a production-grade PostgreSQL cluster:
| Resource | Name | Purpose |
|---|---|---|
| StatefulSet | db | 3-replica PostgreSQL cluster, each with a dedicated PVC |
| Service (Headless) | db | Stable DNS for pod-to-pod communication |
| Job | db-seed | Seeds the users table with initial data once all pods are ready |
The StatefulSet and the seed Job both use a nodeSelector to run on node-01.
All 3 pods and the seed Job are stuck in Pending. No data has been written.
Task
Find why the pods cannot be scheduled and fix the underlying issue so:
- All 3 replicas run on
node-01 - All PersistentVolumeClaims are
Bound - The seed Job completes and the
userstable has data on every pod
Do not modify the StatefulSet, Service, or Job.
Hint 1 — Why are the pods not starting?
A pod in Pending has been accepted by the API server but not yet placed on
any node. Start by reading what the scheduler recorded:
kubectl get pods -n staging
kubectl describe pod -n staging -l app=db
Read the Events section at the bottom carefully. It will tell you what constraint the scheduler evaluated and why no node matched.
Documentation
Hint 2 — Is the scheduler able to place the pod?
The scheduler only places pods on nodes that are Ready. List the nodes and
inspect the one the StatefulSet targets:
kubectl get nodes
kubectl describe node node-01
Check the Conditions section. Look for any condition that is not in its expected state and read the reason field — it will point to what component on the node is failing.
Documentation
Hint 3 — Something on the machine needs attention
The kubelet reports node health to the control plane, but it depends on other
system services to actually run containers. If one of those services is down,
the node goes NotReady even though the kubelet itself is running.
SSH to node-01 and check the status of the system services involved in
container execution:
ssh node-01
systemctl list-units --state=failed
Start any failed service and watch the node recover from cplane-01:
kubectl get nodes --watch
kubectl get pods -n staging --watch
Documentation