CKA Practice: Recover a Broken Static Control-Plane Pod

On a kubeadm cluster the control plane components are not scheduled. The kubelet on cplane-01 runs them directly from manifest files in /etc/kubernetes/manifests/. They are called static pods, and the kubelet owns their whole lifecycle.

That has two consequences worth holding on to:

• The kubelet does not need the API server to run them. If the kubelet is up, it is still trying to run the control plane right now.
• The mirror pods you normally see with kubectl get pods -n kube-system are only read-only copies published to the API. With the API down, they do not exist as far as you are concerned.

So the first question is whether the kubelet is healthy: systemctl status kubelet.

crictl speaks to the container runtime over its socket, so it works with the API server down:

crictl ps            # running containers
crictl ps -a         # also containers that have exited

Compare the two. A component that starts and dies keeps appearing in crictl ps -a with a climbing attempt count, and never appears in crictl ps.

crictl pods is a different list. It shows pod sandboxes, which hold the network and IPC namespaces. The container runs inside the sandbox and has its own ID. The two lists never share an ID, and most crictl subcommands take one or the other, not both. Reach for crictl ps, not crictl pods.

On this playground crictl prints two warnings on every call, because /etc/crictl.yaml does not exist and it falls back to probing the default sockets. It still works. Silence it once:

crictl config --set runtime-endpoint=unix:///run/containerd/containerd.sock

crictl logs <container-id>

logs takes a container ID. Passing it a pod ID from crictl pods, or a pod name, returns NotFound, which reads like the container is gone when it is only the wrong identifier. Take the ID from crictl ps -a, or skip the copying:

crictl logs "$(crictl ps -a --name kube-apiserver -q | head -1)"

A process that exits immediately usually explains itself in its last line.

The kubelet garbage-collects exited containers, so that ID can disappear mid-incident. The logs on disk outlive it:

ls /var/log/pods/kube-system_kube-apiserver-*/
tail -50 /var/log/containers/kube-apiserver-*.log

Know where the kubelet writes container logs. It is the more reliable path.

The component is failing on its own configuration, before it does any real work. Read its static pod manifest and compare it line by line against what the error message named:

grep -n -- '--etcd' /etc/kubernetes/manifests/kube-apiserver.yaml

Edit the manifest in place. You do not need to restart anything by hand.

The kubelet watches /etc/kubernetes/manifests/ and also rescans it on a timer set by fileCheckFrequency, 20 seconds by default, so an edit is picked up shortly after you save it. Changing the file is how you deploy a change to a static pod.

Give it a moment after that. An API server that answers requests is not yet an API server that is ready: /readyz reports failures for a few seconds while its post-start hooks finish.

The control plane pods are back in that listing now. They are mirror pods: the kubelet publishes a read-only copy of each static pod to the API server so that you can see it. The API server does not manage them. The practical difference:

kubectl -n kube-system delete pod kube-apiserver-cplane-01

That command appears to succeed. It deletes the mirror, the kubelet notices, and the mirror comes straight back. It never touched the running container. To change or stop a static pod you edit or move its manifest file, which is what you did in Step 1.

In the exam: kubectl edit on a mirror pod is rejected, and kubectl delete on one is close to useless.