Kubernetes Pod Scheduling: Priority and Preemption

The Kubernetes scheduler places pods in order of priority. When a high-priority pod cannot be scheduled because resources are exhausted, the scheduler preempts lower-priority pods to make room for it.

Pod Priority and Preemption - Kubernetes Docs

Task 1 - Create PriorityClasses

Create two PriorityClass resources: one for critical workloads and one for background workloads.

Steps:

Create PriorityClass named high-priority, value 1000000
Create PriorityClass named low-priority, value 1000

kubectl get priorityclass

Hint: PriorityClass structure

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: <name>
value: <integer>
globalDefault: false

Task 2 - Fill the Nodes

Deploy a filler Deployment with 4 replicas using low-priority. Each pod requests enough CPU to consume most of the available capacity across both worker nodes.

Steps:

Create Deployment named filler, image nginx:alpine, 4 replicas, label app: filler
priorityClassName: low-priority
Resource requests: cpu: 300m

kubectl get pods -l app=filler -o wide

Hint: priorityClassName and resources

spec:
  priorityClassName: <priority-class-name>
  containers:
  - name: <name>
    image: <image>
    resources:
      requests:
        cpu: <cpu-request>

Task 3 - Trigger Preemption

Deploy a critical pod with high-priority and the same cpu: 300m request. The cluster has no room - preemption should evict a low-priority pod to make space.

Steps:

Create pod named critical, image nginx:alpine
priorityClassName: high-priority
Resource requests: cpu: 300m

kubectl get pods -o wide
kubectl get events --sort-by='.lastTimestamp' | grep -i preempt

Watch for a filler pod disappearing as critical gets scheduled. The events output confirms preemption - if it does not appear, the nodes had enough free capacity and no eviction was needed.

Preemption only triggers when no node has enough free CPU to schedule the pod. Each worker node in this challenge has 1 CPU (~950m allocatable after system overhead). Two filler pods at cpu: 300m consume 600m per node, leaving ~100m free - not enough for the critical pod at cpu: 300m, so preemption kicks in. Check actual allocatable CPU with kubectl get nodes -o custom-columns='NAME:.metadata.name,CPU:.status.allocatable.cpu'. For preemption to work, the free CPU remaining on each node after the filler pods land must be less than what the critical pod requests - otherwise the scheduler simply places critical in the free space without evicting anyone.

Hint: pod with priorityClassName

spec:
  priorityClassName: <priority-class-name>
  containers:
  - name: <name>
    image: <image>
    resources:
      requests:
        cpu: <cpu-request>

Task 4 - Priority Without Preemption

A PriorityClass with preemptionPolicy: Never gives a pod high queue priority but prevents it from evicting running pods to make space. Deploy one with a priority higher than everything else in the cluster.

Steps:

Create PriorityClass named batch-priority, value 2000000, preemptionPolicy: Never
Create pod named batch-pod, image nginx:alpine, priorityClassName: batch-priority, requests.cpu: 300m

kubectl get pods

Both batch-pod and the evicted filler pod are Pending. The cluster is full and batch-pod will not preempt critical to get in - despite outranking it.

Now free a slot and watch which pod wins it:

kubectl delete pod critical --grace-period=0 --force

kubectl get pods

batch-pod (priority 2000000) gets the freed slot. The filler pod (priority 1000) stays Pending. preemptionPolicy: Never only disables forced eviction - when a slot opens naturally, priority still decides who gets it first.

Hint: preemptionPolicy field

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: <name>
value: <integer>
preemptionPolicy: Never
globalDefault: false

This challenge is part of the Kubernetes Pod Scheduling skill path.