Challenge ·Easy

CKA Practice: Recover a NotReady Node After a Kubelet Configuration Error

Shiva Swaroop N K
by  Shiva Swaroop N K · on
Kubernetes
A worker node dropped to NotReady and part of the workload went with it. The container runtime is fine and the control plane is healthy; the trail leads from kubectl symptoms down into systemd and the kubelet configuration. Diagnose the node and bring it back.
#cka#kubelet#nodes#troubleshooting

The alert fired eleven minutes ago and has not cleared:

NodeNotReady node-02 condition Ready is Unknown. Workload availability in namespace web degraded: web-canary 0/1 available.

node-02 has become NotReady and the workload running on it is affected. Investigate the node and restore it to a healthy Ready state.

The other nodes and the control plane are healthy, and they should stay that way. Whatever went wrong is local to node-02.

The plan:

  1. Read what the control plane knows, and work out what it cannot know.
  2. Get the kubelet process running again on the node.
  3. Get the node reporting Ready to the API server.
  4. Confirm the workload came back.

Constraints:

  • Repair node-02. Do not delete it, drain it permanently, or replace it.
  • Do not delete or edit the deployments in the web namespace.
  • Do not recreate the cluster.
  • If you cordon the node while you work, uncordon it before you finish.

Three states that look the same from a distance

A NotReady node is a symptom, not a diagnosis. The kubelet passes through three states, and a node can fail at any of them:

  1. The process runs. systemctl reports active. This says nothing about Kubernetes.
  2. The process initialised. The kubelet parsed its configuration, reached the container runtime, and started its control loops.
  3. The node reports Ready. The kubelet posts node status to the API server, and keeps posting it.

Troubleshooting a NotReady node means working out which of the three the node failed to reach. The API server only knows what the node last told it, so once a node stops reporting, the next evidence lives on the node itself.

Pre-flight

Start from the cluster view on cplane-01:

kubectl get nodes -o wide
kubectl describe node node-02
kubectl get pods -n web -o wide

Read the Ready condition and its message before you go anywhere else. Each step below carries its own hints. Open them only as far as you need.

Reading the Ready condition

kubectl describe node node-02 prints the Ready condition with a reason and a message. A node that stopped reporting shows reason: NodeStatusUnknown and message: Kubelet stopped posting node status.

Read that literally. The control plane is not saying the node is broken. It is saying the node went quiet. The node lease in the kube-node-lease namespace stopped being renewed, so after 40 seconds the node controller marked the node Unknown. Nothing on the control plane can tell you why. The next evidence is on the node.

Step 1: Get the kubelet process running

The control plane has told you everything it can. The rest of the evidence is on the node:

ssh node-02
sudo -i

This step passes when systemd reports the kubelet as active. First of the three states, and the one you fix directly.

Asking systemd about the kubelet

The kubelet is a systemd unit on the node, not a pod:

systemctl status kubelet
systemctl is-active kubelet

Read the state word carefully, because three of them mean different things:

  • active (running): the process is up. It may still be failing to do its job.
  • activating (auto-restart): it starts, exits, and systemd restarts it. A crash loop.
  • failed: it exited and systemd gave up restarting it.

activating (auto-restart) is the interesting one. Something makes the kubelet exit very early, every time.

Reading the kubelet journal
journalctl -u kubelet -n 100 --no-pager

In a restart loop the journal repeats the same short block. Find one restart boundary and read the lines above it. Those are the real error. Everything below is systemd starting the process again.

The kubelet loads its configuration file before it contacts the runtime or the API server, so a configuration failure appears at the very top of each attempt and nothing else gets a chance to log.

Where the kubelet's configuration comes from

The kubelet does not read a fixed path. systemd passes it one:

systemctl cat kubelet | grep -- --config

On a kubeadm node the drop-in points at /var/lib/kubelet/config.yaml. That file is a KubeletConfiguration object, and it is strictly decoded, so a value of the wrong type stops startup.

node-01 is healthy and has the same file. Comparing the two is a fast way to see what changed:

ssh node-01 sudo cat /var/lib/kubelet/config.yaml > /tmp/node-01.yaml
diff /tmp/node-01.yaml /var/lib/kubelet/config.yaml

Step 2: Get the node reporting Ready

A running process is not a healthy node. Once the kubelet starts cleanly it re-registers and begins posting status again, which takes a few seconds. Watch it flip from cplane-01:

kubectl get nodes -w

You should not need to do anything else here. If the node stays NotReady after the kubelet has been active for a minute, the process is running but not initialising, which is the second of the three states.

Still NotReady after the kubelet came up

Separate two cases:

  • The kubelet is active but the node never reports. Read the journal again after the restart. A kubelet can start, fail to reach the container runtime or the API server, and keep running while it retries.
  • The node reports Ready but nothing schedules on it. That is a cordon, not a fault. kubectl get nodes prints Ready,SchedulingDisabled, and .spec.unschedulable is true. It is cleared by the inverse of whatever set it.

This challenge judges the node on its Ready condition, so a cordoned but healthy node still passes this step. It will stop you in Step 3, where the workload has to land back on the node.

Step 3: Confirm the workload recovered

A Ready node does not by itself mean the workload recovered. Confirm that too:

kubectl get pods -n web -o wide
kubectl -n web get deployment web-backend web-canary

web-canary is pinned to node-02, so it can only become available once the node is both Ready and schedulable.

More on the same terrain, worth doing alongside this one:

References

How to Author Challenges on iximiuz Labs

Instead of providing a subpar online editing experience, iximiuz Labs offers a helper CLI tool called labctl, allowing you to use your favorite text editor (or a full-featured IDE) to write content from the comfort of your local machine.

Install labctl CLI

curl -sf https://labs.iximiuz.com/cli/install.sh | sh

This will download and install the latest version of the labctl CLI. You only need to do this once per workstation.

Authorize labctl

labctl auth login

This will open a browser window asking you to authorize labctl to access your account. You need to do it after a fresh install of labctl and repeat it whenever the auth session expires.

Pull challenge content

labctl content pull challenge recover-notready-node-kubelet-config-af6617e0

This will create a local copy of the challenge content in a directory named recover-notready-node-kubelet-config-af6617e0. You only need to do this once per challenge.

Stream your changes

labctl content push -fw challenge recover-notready-node-kubelet-config-af6617e0

Run this command in a separate terminal to continuously upload your changes to the server while editing the challenge in your favorite text editor or IDE.

You can also use labctl to create, list, and delete your content. Learn more about the available commands: labctl content --help

Start challenge