Meet the Cast

A few players, one line each:

• stress container: It burns memory or CPU on demand. In this example will grabs 150M of memory to blow past its limit.

stress --vm 1 --vm-bytes 150M --vm-hang 1

• memory limit: the hard cap Kubernetes puts on the container.
• cgroup: the Linux kernel feature that enforces that cap.
• OOM killer: the Linux kernel watchdog that kills a process when its cgroup runs out of memory.
• kubelet: the node agent that restarts the dead container and reports OOMKilled.

The rest of the tutorial is just these five interacting. The playground already runs a broken Deployment called memory-hog. Let's look.

stress --vm 1 --vm-bytes 150M --vm-hang 1

stress --cpu 2

Step 1: Observe the Symptom

List the Pods:

kubectl get pods -l app=memory-hog

You should see something like:

NAME                          READY   STATUS             RESTARTS      AGE
memory-hog-7d4b9c6f4-xxxxx    0/1     CrashLoopBackOff   3 (25s ago)   2m

Notice three things:

• READY 0/1: the container isn't serving.
• RESTARTS keeps going up.
• the status flips between Running, OOMKilled or Error, and CrashLoopBackOff.

Delete the crashing pod and immediately watch its replacement and see it go through the crash/restart cycle. Let it run for a minute or so.

kubectl delete pods -l app=memory-hog &&
kubectl get pods -l app=memory-hog -w
# Press Ctrl+C to stop watching

Step 2: Find Out Why It Crashed

kubectl describe shows the Pod's status and recent events:

kubectl describe pod -l app=memory-hog

Look at the Last State section of the container:

    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137

Step 3: Understand the Mechanism

The container is being OOMKilled because it tries to use more memory than its limit allows.

kubectl describe pod -l app=memory-hog

Under the container, look for its resource block:

    Limits:
      memory:  100Mi
    Requests:
      memory:     50Mi

So the container is capped at 100Mi. If it exceeds that, the kernel's OOM killer terminates it.

In the same output, look at what the container runs:

    Command:
      stress
    Args:
      --vm
      1
      --vm-bytes
      150M
      --vm-hang
      1

stress wants 150M, but the container is capped at 100Mi. The chain:

• requests: used by the scheduler to place the Pod.
• limits: enforced at runtime via the cgroup's memory.max.
• Go over memory.max and the OOM killer kills a process in the cgroup (SIGKILL, exit 137).
• The kubelet marks it OOMKilled and restarts it. That's the loop.

Step 4: Fix It

The app really needs ~150Mi, so give it room:

kubectl set resources deployment memory-hog \
  --limits=memory=256Mi --requests=memory=128Mi

kubectl rollout status deployment/memory-hog

(Or kubectl edit deployment memory-hog and change it by hand.)

Confirm the new Pod is stable:

kubectl get pods -l app=memory-hog

metrics-server is running, so you can check real usage against the limit:

kubectl top pod -l app=memory-hog

NAME                          CPU(cores)   MEMORY(bytes)
memory-hog-xxxxxxxxx-xxxxx    2m           151Mi

It settles around 150Mi, inside the new 256Mi limit and above the old 100Mi cap that kept killing it. That's how you size limits: from real numbers.

until kubectl top pod -l app=memory-hog 2>/dev/null; do
  echo "waiting for metrics..."
  sleep 5
done

Step 5: Quality of Service Classes

requests and limits also give the Pod a Quality of Service (QoS) class. Kubernetes uses it to decide which Pods to evict first when a node runs low on memory.

Check the class of the Pod you just fixed:

kubectl get pod -l app=memory-hog -o jsonpath='{.items[0].status.qosClass}'

Burstable

Three classes, derived from the resource spec:

memory-hog is Burstable: it sets memory but not CPU, and request and limit differ. Set request equal to limit for both CPU and memory to get Guaranteed.

kubectl get pods -A \
  -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,QOS:.status.qosClass'

Step 6: CPU Limits Throttle Instead of Kill

Memory and CPU hit their limits very differently:

• Memory can't be compressed, so over the limit the kernel kills the process (the OOMKill you just saw).
• CPU can be compressed, so over the limit the kernel just throttles it. The container keeps running, only slower.

So a tight CPU limit never causes OOMKilled. It quietly makes the app slow, which is harder to spot. Deploy a CPU hog with a limit below what it wants:

kubectl apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cpu-hog
  labels:
    app: cpu-hog
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cpu-hog
  template:
    metadata:
      labels:
        app: cpu-hog
    spec:
      containers:
        - name: stress
          image: polinux/stress
          command: ["stress"]
          args: ["--cpu", "1"]
          resources:
            requests:
              cpu: "100m"
            limits:
              cpu: "500m"
EOF

kubectl rollout status deployment/cpu-hog

stress --cpu 1 wants a whole core (1000m), but the limit is 500m (half a core). The Pod stays Running, no crash:

kubectl get pods -l app=cpu-hog

Check the cgroup CPU stats to see the throttling:

POD=$(kubectl get pod -l app=cpu-hog -o jsonpath='{.items[0].metadata.name}')
kubectl exec "$POD" -- cat /sys/fs/cgroup/cpu.stat

usage_usec 4123456
user_usec 4011234
system_usec 112222
nr_periods 412
nr_throttled 410
throttled_usec 38221456

The fields that matter:

• nr_periods: CPU accounting periods passed.
• nr_throttled: periods the container was throttled.
• throttled_usec: microseconds spent waiting for CPU.

nr_throttled is ~99% of nr_periods here: the limit is starving the workload. Same fix as before, raise the limit. But nothing crashed, so you only find it by looking.

kubectl top shows it pinned at the limit:

kubectl top pod -l app=cpu-hog

NAME                       CPU(cores)   MEMORY(bytes)
cpu-hog-xxxxxxxxx-xxxxx    500m         1Mi

Flat at 500m with rising nr_throttled: that's the throttling signature.

Compare with a Pod that isn't throttled

memory-hog has no CPU limit, so it's never throttled. Read its cpu.stat for contrast:

POD=$(kubectl get pod -l app=memory-hog -o jsonpath='{.items[0].metadata.name}')
kubectl exec "$POD" -- cat /sys/fs/cgroup/cpu.stat

usage_usec 1820345
user_usec 1700123
system_usec 15982
nr_periods 0
nr_throttled 0
throttled_usec 0

No CPU limit means no quota, so the kernel does no CFS accounting at all: nr_periods is 0 and nothing is ever throttled.

The cpu-hog container racks up periods and gets throttled in nearly all of them. The memory-hog container has no limit, so it has no periods to throttle. That contrast is how you tell a throttled container from a healthy one.

NODE=$(kubectl get pod -l app=cpu-hog -o jsonpath='{.items[0].spec.nodeName}')
kubectl get --raw "/api/v1/nodes/$NODE/proxy/metrics/cadvisor" \
  | grep container_cpu_cfs_throttled_periods_total | grep cpu-hog

container_cpu_cfs_throttled_periods_total{container="stress",...,pod="cpu-hog-..."} 117

Step 7: What Happens Between Request and Limit?

You've seen the two extremes: at the limit, memory gets you killed and CPU gets you throttled. But what about the band between request and limit, where most containers actually live? The answer is different for the two resources.

First, what each value means:

• request: what the scheduler reserves for you. It's your guaranteed floor.
• limit: the hard ceiling the kernel enforces.

CPU in the band

CPU requests set your share of the CPU (the cgroup's cpu.weight). Usage between request and limit is completely fine:

• The node is idle: you run freely, all the way up to your limit. Nothing throttles you below it.
• The node is busy: every container is guaranteed at least its request. Above your request you only get spare CPU that nobody else is using, so you may slow down, but you're still not throttled by the kernel. Throttling only happens at the limit.

So in the CPU band you are never killed and never throttled. The worst case under contention is that you simply don't get more than you requested.

Memory in the band

Memory requests are not enforced at runtime, only used for scheduling. Usage between request and limit:

• The node has memory to spare: you run fine, right up to your limit.
• The node is under memory pressure: the kubelet starts evicting Pods to reclaim memory. Pods using more than their request go first (BestEffort, then over-request Burstable). Eviction is graceful: the Pod is deleted and rescheduled elsewhere, not SIGKILLed in place.

Crossing your own limit always gets you OOMKilled by the kernel. But there's a second way to be killed below your limit: if the node fills up faster than the kubelet can evict, the kernel's node-level OOM killer fires to save the node and SIGKILLs a process right away. It picks the victim by oom_score, derived from QoS and how far over its request a container is:

• A container above its request is a prime target.
• A container below its request is protected (negative oom_score_adj).

So below your limit but above your request, under sudden pressure you can be either evicted (graceful) or OOMKilled by the node (abrupt). Staying under your request is what truly keeps you safe.

The whole picture

The one-line version: CPU is compressible, so the worst that happens is you slow down; memory is not, so crossing the limit kills you, and crossing your request makes you an eviction target.

Step 8: Enforce Defaults with LimitRange

A LimitRange fills in default requests and limits for containers that skip them, and enforces min/max bounds. Apply one to the default namespace:

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: LimitRange
metadata:
  name: default-resources
spec:
  limits:
    - type: Container
      default:            # applied as the container's LIMIT if unset
        cpu: "200m"
        memory: "128Mi"
      defaultRequest:     # applied as the container's REQUEST if unset
        cpu: "100m"
        memory: "64Mi"
      max:                # a container may not request/limit more than this
        cpu: "1"
        memory: "512Mi"
EOF

Now deploy a Pod that sets no resources at all:

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: no-resources
spec:
  containers:
  - name: nginx
    image: nginx
EOF

The manifest sets no resources, but the LimitRange fills them in. Use kubectl describe to see what the Pod actually got, the same way you inspected the crashing Pod earlier:

kubectl describe pod no-resources

Look at the container's Limits and Requests:

    Limits:
      cpu:     200m
      memory:  128Mi
    Requests:
      cpu:        100m
      memory:     64Mi

It asked for nothing, yet came up Burstable with the defaults filled in, not BestEffort. Confirm:

kubectl get pod no-resources -o jsonpath='{.status.qosClass}'

max is enforced too: a Pod asking for memory: 1Gi here is rejected, over the 512Mi cap.

Step 9: Takeaways

• CrashLoopBackOff is a symptom. Check lastState.terminated.reason and events.
• OOMKilled + exit 137: the OOM killer enforced the memory limit (memory.max).
• requests place the Pod; limits are enforced at runtime.
• Over the memory limit, you're killed. Over the CPU limit, you're throttled (check cpu.stat), never killed.
• Between request and limit: CPU is never throttled (just loses spare cycles under contention); memory can be evicted or, under sudden node pressure, OOMKilled by the node even below your limit. Staying under your request is what keeps you safe.
• QoS class (Guaranteed/Burstable/BestEffort) comes from requests and limits and sets eviction order.
• Use a LimitRange so no Pod lands in BestEffort by accident.
• Size limits from real usage and leave room for spikes.
• If memory grows without bound, a bigger limit just delays the crash. Fix the leak.

Practice

Now put it into practice. These two challenges drop you into a broken cluster and ask you to apply exactly what you just learned: