Resource metrics with kubectl top

Events tell you what happens. Metrics tell you what gets consumed. They are different questions and they need different tools.

In this cluster there are three Pods waiting for you, and their names are a spoiler:

kubectl get pods -l escenario=metricas

api: burns CPU non-stop in an infinite loop.
holgazan (Spanish for slacker): reserves 800m of CPU and half a gig of RAM, and does absolutely nothing.
anonimo: declares neither requests nor limits. As far as the scheduler is concerned, it does not exist.

Step 1: Where do the metrics come from?

Kubernetes does not store metrics. The API server does not know how much CPU a Pod consumes, and it does not care: its job is to store the desired state, not to measure the world.

What it does have is a mechanism for another component to serve that information as if it were part of the API. It is called API Aggregation, and it looks like this:

kubectl get apiservice v1beta1.metrics.k8s.io

kubectl get apiservice v1beta1.metrics.k8s.io -o yaml

Notice spec.service: the metrics.k8s.io group is not served by the kube-apiserver. It is served by a Service, in a namespace, like any other application. The API server proxies to it.

That application is metrics-server, and how it works is humbler than people assume:

Every 15 seconds, it asks every kubelet in the cluster.
Each kubelet returns the CPU and memory consumption of its containers (which it in turn gets from the kernel's cgroups, via cAdvisor).
metrics-server keeps it in memory.

There is no database. There is no history. If you restart metrics-server, everything is lost, and it only knows what has happened in the last few seconds.

Note

💡 This explains an error you will see in real clusters: error: Metrics API not available. It is not a kubectl failure nor a Kubernetes failure: it is that nobody has installed metrics-server. Many distributions (k3s among them) ship it out of the box; kubeadm, for instance, does not.

Step 2: `kubectl top`

Now that you know where the data comes from, measure it:

kubectl top nodes

kubectl top pods

kubectl top pods --containers

kubectl top pods --sort-by=cpu

api is pinned at its CPU limit. holgazan and anonimo consume crumbs.

Save the list sorted by CPU:

kubectl top pods --sort-by=cpu > /home/laborant/consumo.txt
cat /home/laborant/consumo.txt

Step 3: The three numbers that are not the same number

Here is the lesson of this unit, and it is the one that saves the most money in the whole course.

For a single container there are three different numbers, and mixing them up is the most expensive and most common mistake in Kubernetes:

	What it is	Who uses it	Where you see it
`requests`	What the Pod reserves	The scheduler, to decide whether it fits on a node	`kubectl describe pod`
`limits`	What the Pod may get to use	The kubelet and the kernel: CPU throttling, memory OOMKill	`kubectl describe pod`
actual usage	What the Pod is consuming right now	Nobody. It is information, not a rule	`kubectl top`

And now the important part: the node fills up with the requests, not with actual usage.

Check it. Look at what the node has reserved:

kubectl describe node | grep -A12 "Allocated resources"

And compare it with what is really used:

kubectl top node

kubectl top pods

The two numbers look nothing alike. holgazan has 800m of CPU reserved and consumes practically nothing: those 800m are set aside for it, and no other Pod can use them. The node can reject a new Pod for lack of CPU while sitting at 5% actual usage.

This is, exactly, why a cluster can be full and empty at the same time. And why the VPA exists.

Note

⚠️ Watch out for the opposite extreme: the Pod anonimo reserves nothing. For the scheduler it takes up zero, so it fits anywhere, always. Sounds great until the node gets saturated and the kubelet has to evict Pods: with no requests, its QoS class is BestEffort and it is the first to fall. Not setting requests is not saving: it is giving up every guarantee.

Step 4: Fix the slacker

You have the number. Now use it: create a holgazan-ajustado with the requests that kubectl top says it really needs.

cat << 'EOF' > holgazan-ajustado.yaml
apiVersion: v1
kind: Pod
metadata:
  name: holgazan-ajustado
spec:
  containers:
  - name: web
    image: ghcr.io/iximiuz/labs/nginx:alpine
    resources:
      requests:
        cpu: "20m"        # <- what kubectl top really says
        memory: "32Mi"
      limits:
        cpu: "200m"       # headroom for spikes
        memory: "128Mi"
EOF

kubectl apply -f holgazan-ajustado.yaml
kubectl describe node | grep -A12 "Allocated resources"

Notice what just happened: you have freed 780m of CPU on the node without changing a single line of the application. That is the unglamorous, very profitable work of rightsizing.

Step 5: The limits of `kubectl top`

kubectl top is the right tool for three questions:

What is consuming right now?
Which Pod has run away?
Do my requests resemble reality?

And it is completely useless for these others:

What happened last night at 3:00?
Is this consumption normal for a Monday?
How many requests per second does my API serve? (kubectl top knows nothing about your application: it only sees CPU and memory)
Warn me when something goes wrong.

Four questions, a single gap: there is no history, there are no application metrics and there are no alerts. That is exactly what you are going to set up in the next lesson.

Summary

Kubernetes does not store metrics. The metrics.k8s.io API is served by metrics-server, aggregated into the cluster's API, and it keeps the data in memory: no history.
Without metrics-server there is no kubectl top, and there is no HPA or VPA either.
requests ≠ limits ≠ actual usage. The node fills up with the requests. A cluster can be full for the scheduler and empty in reality.
Adjusting the requests to actual usage (rightsizing) frees capacity without touching the application. It is what the VPA automates.
kubectl top answers "what is happening now". For "what happened last night" and "warn me when" you need something else.

Course progress