Troubleshoot Why the Calendar Todo App Pods Are Stuck in Pending
Scenario
You are the Kubernetes administrator for the Calendar Todo — a productivity web application that lets users manage tasks and schedules.
The application has been deployed to the prod namespace as
calendar-todo-app, but something went wrong during the rollout. All 3 replicas
are stuck in Pending state and users cannot access the app.
Task
- Diagnose why the pods cannot schedule or start
- Fix the underlying infrastructure issue
- Verify the application is fully operational and serving content
Hint 1 — Check Why the Pods Are Pending
Pods stuck in Pending means the scheduler cannot find a node to place them
on. Start by checking the pod events to understand what the scheduler is
reporting:
kubectl get pods -n prod
kubectl describe pod -n prod -l app=calendar-todo-app
kubectl get events -n prod | grep -i warning
Documentation
Hint 2 — Check the Node Status
Once you find a scheduling error in the events, check the status of all nodes.
A NotReady node cannot accept any pods, even if the pod's tolerations and
node selectors are correct.
kubectl get nodes
kubectl describe node node-01
Look at the Conditions section. All conditions showing Unknown with the
message Kubelet stopped posting node status means kubelet on that node has
gone silent.
Documentation
Hint 3 — Diagnose the Kubelet on node-01
SSH to node-01 and check the kubelet service status and logs:
sudo systemctl status kubelet
sudo journalctl -u kubelet | grep -i error
If kubelet is crashing on startup, the logs will tell you exactly why. A
common cause is a missing or corrupt file in /var/lib/kubelet/pki/.
kubelet maintains a serving certificate pair (kubelet.crt and kubelet.key)
that it uses to serve its own HTTPS endpoints. If one file is present without
the other, kubelet treats the pair as broken and refuses to start. Check what
files are present:
ls -la /var/lib/kubelet/pki/
Documentation
Hint 4 — Think About Node Lifecycle
A node is not permanent. Just like pods can be deleted and recreated, a node can be removed from the cluster and re-registered. When a node's state is too broken to repair in place, the right move is to treat it like a fresh machine.
Sometimes a node reaches a state where its local Kubernetes files, configurations, or credentials are broken or incomplete. In that case, trying to patch individual files is risky and unreliable. The safer approach is to wipe the node's Kubernetes state entirely and let it re-register with the cluster from scratch — so the cluster itself can provision everything the node needs to operate correctly.
Think about the full lifecycle of a node:
- How did
node-01join the cluster in the first place? - What command registers a node with the control plane?
- Before re-registering, what needs to be cleaned up first?
kubeadm has subcommands for both sides of this — wiping a node's
existing Kubernetes state and joining it back to the cluster.
Look into:
kubeadm reset --help
kubeadm token --help
kubeadm join --help
Documentation