kubelet

kubelet is the primary node agent that runs on every worker node in a Kubernetes cluster. As one of the core components that makes Kubernetes work, it acts as the bridge between the Kubernetes control plane and the container runtime on each node.

kubelet operates in a continuous reconciliation loop:

1. Watches for Pod specifications from the API server
2. Compares the desired state (what should be running) with the actual state (what is running)
3. Takes action to bring the actual state in line with the desired state
4. Reports the current status back to the control plane

In essence, kubelet is the "hands and feet" of Kubernetes on each node— it takes declarative Pod specifications from the control plane and makes them a reality by managing containers on the worker nodes.

However, kubelet doesn't run containers directly. Instead, it delegates this responsibility to a container runtime like containerd or CRI-O. To support multiple runtimes and provide flexibility in choosing the runtime for your environment, kubelet communicates with these runtimes via the Container Runtime Interface (CRI).

kubelet in a Kubernetes cluster

Follow these steps to install kubelet:

Set the version to install:

KUBE_VERSION=v1.33.3

KUBE_VERSION=$(curl -fsL https://dl.k8s.io/release/stable.txt)

Download and install kubelet from the official Kubernetes releases:

curl -fsSLO https://dl.k8s.io/${KUBE_VERSION?}/bin/linux/amd64/kubelet
sudo install -m 755 kubelet /usr/local/bin

Download the systemd unit file to run kubelet as a systemd service:

sudo wget -O /etc/systemd/system/kubelet.service https://labs.iximiuz.com/content/files/courses/kubernetes-the-very-hard-way-0cbfd997.beta/02-worker-node/02-kubelet/__static__/kubelet.service?v=1756470611

With kubelet installed, you can now configure it.

For now, the only configuration needed is specifying how kubelet should communicate with the container runtime:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration

containerRuntimeEndpoint: unix:///var/run/containerd/containerd.sock
cgroupDriver: systemd

sudo mkdir -p /var/lib/kubelet/config.d
sudoedit /var/lib/kubelet/config.d/99-cri.conf

Start the kubelet service:

sudo systemctl daemon-reload
sudo systemctl enable --now kubelet

Congratulations! You have successfully installed kubelet. 🎉

kubelet exposes an HTTP API endpoint (typically on port 10250) that allows the Kubernetes API server and other components to interact with it.

This endpoint provides access to:

• Pod logs and exec sessions
• Node metrics and health information

In production environments, this endpoint must be secured through proper authentication and authorization mechanisms.

kubelet authentication and authorization

To simplify the setup for this lesson, you'll disable kubelet authentication and authorization:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration

authentication:
    anonymous:
        enabled: true
    webhook:
        enabled: false

authorization:
    mode: AlwaysAllow

sudoedit /var/lib/kubelet/config.d/70-authnz.conf

Restart the kubelet service to apply the configuration changes:

sudo systemctl restart kubelet

Verify that the kubelet API endpoint is accessible:

curl -k https://localhost:10250/healthz

Static pods are pods managed directly by kubelet on a specific node rather than by the Kubernetes API server.

Unlike regular pods created and managed through the cluster's control plane, static pods are defined by placing pod manifest files in a directory that kubelet monitors.

When kubelet finds a pod manifest in the static pod directory, it automatically creates and manages that pod. If the pod crashes or stops, kubelet automatically restarts it.

kubelet also creates a mirror pod in the Kubernetes API server for each static pod. This mirror pod allows you to see the static pod when you run kubectl get pods, but you cannot control the static pod through the API server—only kubelet can manage it directly.

Static pod functionality is disabled by default. To enable it, you need to configure kubelet.

Create the static pod directory:

sudo mkdir -p /etc/kubernetes/manifests

Configure kubelet to monitor the static pod directory by creating a configuration file in the kubelet drop-in directory:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration

staticPodPath: /etc/kubernetes/manifests

sudoedit /var/lib/kubelet/config.d/50-static-pods.conf

Restart the kubelet service to apply the configuration changes:

sudo systemctl restart kubelet

Create your first static pod by deploying a simple application called podinfo to verify everything works correctly.

Create the pod manifest:

apiVersion: v1
kind: Pod
metadata:
    name: podinfo
spec:
    hostNetwork: true
    containers:
        - name: podinfo
          image: ghcr.io/stefanprodan/podinfo:latest
          ports:
              - containerPort: 9898

sudoedit /etc/kubernetes/manifests/podinfo.yaml

Verify that podinfo is running by checking if it appears in the list of pods:

curl -sfk https://localhost:10250/pods | jq

Verify that the podinfo application is working correctly:

curl http://localhost:9898

Even though the pod is running, it won’t show up when you list containers directly:

sudo nerdctl ps

sudo ctr container ls

This happens because containerd organizes containers into namespaces (similar to Kubernetes), and Kubernetes uses the k8s.io namespace by default.

You can verify this by running:

sudo ctr namespace ls

To see containers running in the k8s.io namespace, use the --namespace flag:

sudo nerdctl ps --namespace k8s.io

sudo ctr --namespace k8s.io container ls

kubeletctl is a command-line tool that provides direct access to the kubelet API.

It is useful for debugging and inspecting kubelet instances, querying pod information, retrieving logs, and exploring the kubelet's internal state without requiring a full Kubernetes cluster setup.

Follow these steps to install kubeletctl:

Set the version to install:

KUBELETCTL_VERSION=v1.13

KUBELETCTL_VERSION=$(curl -s https://api.github.com/repos/cyberark/kubeletctl/releases/latest | jq -r .tag_name)

Download and install kubeletctl from GitHub:

curl -fsSLO https://github.com/cyberark/kubeletctl/releases/download/${KUBELETCTL_VERSION?}/kubeletctl_linux_amd64
sudo install -m 755 kubeletctl_linux_amd64 /usr/local/bin/kubeletctl

With kubeletctl installed, you can perform various operations:

Check the merged configuration:

kubeletctl configz

List all pods:

kubeletctl pods

Return container logs:

kubeletctl containerLogs -p podinfo-worker -c podinfo

Execute a command in a container:

kubeletctl exec "ls /" -p podinfo-worker -c podinfo

In this lesson, you learned about kubelet, the primary node agent that runs on every worker node in a Kubernetes cluster.

Key takeaways:

• Control plane bridge: kubelet acts as the essential link between the Kubernetes control plane and the container runtime, translating API server instructions into container operations
• Reconciliation loop: Continuously watches for pod specifications, compares desired vs actual state, and takes corrective actions to maintain cluster consistency
• CRI delegation: Doesn't run containers directly but communicates with container runtimes like containerd through the Container Runtime Interface (CRI)
• Static pods: Enables running critical cluster components directly on nodes before the control plane becomes available, with automatic restart capabilities
• API endpoint: Provides an HTTPS interface on port 10250 for log access, metrics, and container execution—essential for cluster operations

With kubelet now running and managing static pods, you have established the node-level foundation for Kubernetes workload management.

• Container Runtimes
• Container Runtime Interface (CRI)
• Kubelet authn/authz
• Static pods
• kubeletctl
• cri-tools