Priority and preemption

Until now, when a Pod did not fit, it stayed in Pending and that was the end of the story. Polite, but naive: if the Pod that does not fit is the payment gateway and what is occupying the node is a reports job that can wait until tomorrow, someone has made the wrong decision.

Priority is how you tell the scheduler that not all Pods are worth the same. And preemption is what it does with that information: when an important Pod fits nowhere, the scheduler looks for a node where, by kicking out less important Pods, it would fit. And it kicks them out.

The playground has already worked out the exact size so that only one of these Pods fits on node-01, and has left the two manifests in /home/laborant/manifests. Look at them from the dev-machine tab:

cat /home/laborant/manifests/reports.yaml

cat /home/laborant/manifests/api.yaml

kubectl get node node-01 -o yaml | yq .status.allocatable

They are identical except for the name. Your job is to give them different priorities and see what happens.

Step 1: The PriorityClasses

Create prioridades.yaml:

cat << 'EOF' > prioridades.yaml
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: baja
value: 100
globalDefault: false
description: "Cargas que pueden esperar: informes, procesos por lotes, experimentos."
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: alta
value: 1000000
globalDefault: false
preemptionPolicy: PreemptLowerPriority
description: "Servicios críticos de negocio."
EOF

The two classes keep their Spanish names and descriptions, as in the book: baja (low) is for "workloads that can wait: reports, batch jobs, experiments", and alta (high) is for "business-critical services".

The YAML, explained in questions and answers

Does a PriorityClass have a namespace?

No. It is a cluster-scoped object, like StorageClasses, IngressClasses or nodes. It makes sense: priority is a shared scale, and it would be useless for each team to have its own.

What is the value and what scale is used?

An integer, and the higher, the more important. You make up the scale yourself, but there is a reserved range: above one billion (1000000000) sit the system classes, system-cluster-critical and system-node-critical, used by Kubernetes's own components. Look at them, they already exist in your cluster:

kubectl get priorityclass

What happens if I set globalDefault: true?

Every Pod that declares no priority inherits that class. There can be only one in the whole cluster, and it is a serious decision: making "critical" the default is exactly as useless as having no priorities at all. If there is no default class, the priority of a Pod with no class is zero.

And preemptionPolicy?

It decides whether this class evicts or only jumps the queue. With PreemptLowerPriority (the default), a Pod that does not fit can kick others out. With Never, the Pod gets first pick when there is room, but it does not evict anyone: it goes to the front of the queue and waits. That second option is very useful for important but not urgent workloads, and it is the one that avoids scares.

kubectl apply -f prioridades.yaml
kubectl get priorityclass

Step 2: Fill the node with the expendable

Edit /home/laborant/manifests/reports.yaml and add a single line inside its spec:

  priorityClassName: baja

And apply it:

kubectl apply -f manifests/reports.yaml
kubectl get pod reports -o wide

It is on node-01, perfectly calm, taking up 70% of the node's CPU. Nobody bothers it.

Step 3: The important one arrives

Now edit /home/laborant/manifests/api.yaml with the other class:

  priorityClassName: alta

And get ready to watch two things at once. Apply and observe:

kubectl apply -f manifests/api.yaml
kubectl get pods -o wide --watch

Within seconds: the Pod reports disappears and the Pod api goes from Pending to Running on node-01.

What just happened, step by step, and it is worth understanding it well:

1. The scheduler tries to place api. It does not fit on node-01 (the only node its nodeSelector allows) because reports has the CPU reserved.
2. Instead of giving up, it evaluates preemption: it looks for which lower-priority Pods it would have to kick off that node to make room.
3. It picks reports as the Pod to evict and evicts it, respecting its grace period (and, as far as possible, its PodDisruptionBudgets).
4. With the room freed up, it schedules api.

Read it in the events, which is where the confession is left:

kubectl get events --sort-by=.lastTimestamp | tail -10

kubectl describe pod api | grep -A5 Events

You will see the reason Preempted and the name of the evicted Pod.

The fine print you need to know

The evicted Pod does not come back. reports was a bare Pod, so it is gone for good. Had it been part of a Deployment, the ReplicaSet would create a replacement that would go to another node, or stay in Pending waiting for room. The moral: preemption does not reschedule the evicted Pod, it only evicts it; whoever recreates it, if anyone, is its controller.

Priority also orders the queue. It is not only for evicting: among several pending Pods, the scheduler serves the higher-priority ones first, even when nobody has to be kicked out.

Careful with priority as a weapon. In a shared cluster, if every team can create PriorityClasses and use them freely, everyone will end up critical and you will be back at square one (with the disadvantage of having added complexity). That is why PriorityClasses are governed: their creation is restricted with RBAC and their use is limited with ResourceQuotas and scopeSelector, which let you say "in this namespace you can only have 2 high-priority Pods".

Don't confuse priority with QoS. They are two different hierarchies and act at different moments: priority is used by the scheduler to decide whom to place and whom to kick out; the QoS class is used by the kubelet to decide whom to evict when the node runs out of memory. A Pod can be high priority and BestEffort at the same time, and be the first to fall when the node is short of memory.

Summary

• A PriorityClass is a cluster-scoped object with an integer value: the higher, the more important.
• Preemption evicts lower-priority Pods to make room for a higher-priority one. preemptionPolicy: Never prioritizes without evicting.
• The evicted Pod does not reschedule itself: its controller recreates it, or nobody does.
• There can be only one globalDefault, and without it the default priority is zero.
• Priority (scheduler, where) and QoS (kubelet, whom to kill) are different scales. They do not mix.