Requests, limits and QoS classes

In the first lesson you saw, without understanding it, one line of the describe: QoS Class: BestEffort. Now it is time to understand it, because behind those two words is the contract your Pod signs with the cluster, and that contract decides two serious things: where you get scheduled and who gets killed first when the node runs out of memory.

Two fields, two very different meanings:

• requests: what the Pod asks for. It is the only thing the scheduler looks at to decide whether it fits on a node. It is not real consumption: it is a reservation.
• limits: the ceiling. It is what the node's kernel watches while the container runs.

And here is the detail most people get wrong: the scheduler does not look at the limits, and the kernel does not look at the requests. They are two worlds that do not even talk to each other.

Work from the dev-machine tab.

Step 1: The three classes, in a single file

Kubernetes does not ask you which class you want: it infers it from what you write. Create qos.yaml with the three cases:

cat << 'EOF' > qos.yaml
apiVersion: v1
kind: Pod
metadata:
  name: reports
spec:
  containers:
  - name: app
    image: ghcr.io/iximiuz/labs/nginx:alpine
---
apiVersion: v1
kind: Pod
metadata:
  name: api
spec:
  containers:
  - name: app
    image: ghcr.io/iximiuz/labs/nginx:alpine
    resources:
      requests:
        cpu: 50m
        memory: 64Mi
      limits:
        cpu: 200m
        memory: 128Mi
---
apiVersion: v1
kind: Pod
metadata:
  name: db
spec:
  containers:
  - name: app
    image: ghcr.io/iximiuz/labs/nginx:alpine
    resources:
      requests:
        cpu: 100m
        memory: 128Mi
      limits:
        cpu: 100m
        memory: 128Mi
EOF

The YAML, explained in questions and answers

What does 100m of CPU mean?

One hundred millicores. A millicore is a thousandth of a core, so 100m is a tenth of one. CPU is a compressible resource: it can be handed out in slices of time. 1 is a whole core; 500m is half of one.

And isn't 128Mi the same as 128M?

No, and the difference costs you. Mi is a mebibyte (1024 squared, binary base); M is a megabyte (one million, decimal base). Kubernetes accepts both, but mixing them in the same capacity spreadsheet is an endless source of mismatches. Always use Mi and Gi.

How exactly is the QoS class computed?

With three rules, in this order:

1. Guaranteed: every container in the Pod declares requests and limits that are equal, for both CPU and memory. Not a single one may be missing.
2. Burstable: at least one container declares some request or limit, but the rule above is not met.
3. BestEffort: nobody declares anything.

Notice how strict the first one is: it is enough for one container out of five in a Pod to have its CPU limit one millicore above its request for the whole Pod to drop to Burstable.

What is the class for, if my Pods already have the resources I asked for?

When a node runs out of memory, the kubelet has to evict Pods to survive, and it evicts them by class: first the BestEffort ones (they promised nothing, they are owed nothing), then the Burstable ones that are consuming above their requests, and last the Guaranteed ones. Your QoS class is, literally, your place in the eviction queue.

Apply it and ask the API for the verdict:

kubectl apply -f qos.yaml
kubectl get pods -o custom-columns=POD:.metadata.name,QOS:.status.qosClass

Three Pods, three classes, and you have not written the word "QoS" anywhere.

Step 2: Going over the CPU limit and going over the memory limit are not the same thing

Here is the asymmetry you have to carry with you. Create tragamemoria.yaml (tragamemoria means "memory eater"):

cat << 'EOF' > tragamemoria.yaml
apiVersion: v1
kind: Pod
metadata:
  name: tragamemoria
spec:
  containers:
  - name: app
    image: ghcr.io/iximiuz/labs/nginx:alpine
    command: ["sh", "-c", "echo empiezo a comer memoria; tail /dev/zero"]
    resources:
      requests:
        cpu: 50m
        memory: 32Mi
      limits:
        cpu: 50m
        memory: 32Mi
EOF

That tail /dev/zero is a classic trick: it reads infinite zeros and keeps piling them up in memory, so the container grows nonstop until it hits its 32Mi ceiling. (The echo says "I'm starting to eat memory".)

kubectl apply -f tragamemoria.yaml
kubectl get pod tragamemoria --watch

That --watch keeps listening: as soon as you see the RESTARTS column go up, exit with Ctrl-C and ask for the real reason behind the cut:

kubectl get pod tragamemoria -o jsonpath='{.status.containerStatuses[0].lastState}' | jq .

OOMKilled. The node's kernel does not negotiate: when a container exceeds its memory limit, it kills it. And since the default restartPolicy is Always, the kubelet restarts it, it eats memory again, it dies again, and you end up in the CrashLoopBackOff you already know, but with a new cause in your catalog.

Compare it with what happens when you go over the CPU:

What happens to a container that wants more CPU than its limit?

Nothing dramatic: it gets throttled. The kernel simply gives it less processor time, so the application runs slowly, but it stays alive. CPU is compressible, memory is not. That is why:

• A badly sized CPU limit shows up as unexplained latency, and it is extremely hard to diagnose if you do not have it on your radar (look at the container's throttling metric).
• A badly sized memory limit shows up as OOMKilled, which at least is honest and appears in the describe.

Summary

• requests = what gets reserved and the only thing the scheduler looks at. limits = the ceiling the kernel watches.
• The QoS class is not declared, it is inferred: Guaranteed (requests == limits in everything), Burstable (something declared), BestEffort (nothing).
• The class decides the eviction order when the node runs out of memory. BestEffort goes first.
• Going over the CPU: throttling, the application runs slowly. Going over the memory: OOMKilled, the application dies.
• Without requests there is no sensible scheduling, no fair quotas and no HPA (you will see it in the Scaling module).