ResourceQuotas and LimitRanges

With RBAC you decide who can create things in a namespace. With Pod Security you decide what those things can be like. The third question is missing, the one that ends in an argument with the team next door: how much they can consume.

In the book's Policies chapter, ResourceQuota and LimitRange appear in consecutive sections. You are going to discover the reason here in the worst possible way: by trying to create a Pod.

Two objects share the work, and they only make sense together:

• The ResourceQuota puts a ceiling on the total consumption of the namespace.
• The LimitRange sets defaults, minimums and maximums for each individual container.

The playground comes with the tienda namespace already created. Work from the dev-machine tab.

Step 1: The ResourceQuota

Create quota.yaml:

cat << 'EOF' > quota.yaml
apiVersion: v1
kind: ResourceQuota
metadata:
  name: cuota-tienda
  namespace: tienda
spec:
  hard:
    pods: "5"
    requests.cpu: "1"
    requests.memory: 1Gi
    limits.cpu: "2"
    limits.memory: 2Gi
EOF

The YAML, explained in questions and answers

What does hard mean?

That they are hard limits: the API rejects on the spot any creation that exceeds them. There are no warnings and no margin, the object simply is not created.

What exactly does requests.cpu: "1" add up?

The sum of the CPU requests of all the Pods in the namespace cannot exceed one core. Here the circle closes with the QoS lesson: requests do not only guide the scheduler, they are also the currency the quota charges in. And limits.cpu caps the sum of the ceilings, which is what the namespace could come to consume at a peak.

Can I limit other things besides compute and Pods?

Yes: the number of Services, of Secrets, of ConfigMaps, of PVCs, total gigabytes of requested storage, and even Services of type LoadBalancer (useful, because each one costs real money in a cloud). A quota is the namespace's overall budget, not just its CPU budget.

And if I want to charge differently depending on the kind of workload?

That is what scopes and scopeSelector are for: a quota can apply only to the Pods of a specific PriorityClass, or only to those that are not terminating (NotTerminating, that is, the long-running ones as opposed to Jobs). It is how you reserve capacity for what is critical without choking the rest.

Apply it and look at the state of the budget:

kubectl apply -f quota.yaml
kubectl describe resourcequota cuota-tienda -n tienda

The Used column against the Hard column: that is how you read a quota.

Step 2: The rejection that surprises (and why the LimitRange is not optional)

Before going on, create the most innocent Pod in the world in the governed namespace (sin-recursos means "no resources"):

kubectl run sin-recursos --image=ghcr.io/iximiuz/labs/nginx:alpine -n tienda

Rejected. The message is explicit: must specify limits.cpu, requests.cpu.... And this surprises everyone the first time, so it helps to understand it well:

Why does the quota reject a Pod that asks for nothing?

Because a quota that limits requests.cpu needs to know how much each Pod adds up to. A Pod with no declared requests is a Pod that cannot be accounted for, so the API prefers to reject it rather than leave a hole in the budget. Practical consequence: the moment you set a compute quota, declaring resources stops being optional for everyone in that namespace, and every YAML that did not do so stops working all at once.

That is exactly the incident the LimitRange exists to prevent.

Step 3: The LimitRange

Create limitrange.yaml:

cat << 'EOF' > limitrange.yaml
apiVersion: v1
kind: LimitRange
metadata:
  name: limites-tienda
  namespace: tienda
spec:
  limits:
  - type: Container
    defaultRequest:
      cpu: 100m
      memory: 64Mi
    default:
      cpu: 200m
      memory: 128Mi
    max:
      cpu: "1"
      memory: 1Gi
    min:
      cpu: 10m
      memory: 16Mi
EOF

The YAML, explained in questions and answers

What is the difference between defaultRequest and default?

Confusing names for a pair you already know: defaultRequest is the default request and default is the default limit. They are applied, at admission, to every container that does not declare its own.

What does type: Container refer to?

The scope of the rules. Container is the usual one. There are also Pod (where the minimums and maximums apply to the sum of its containers) and PersistentVolumeClaim (to bound the size of the storage someone can claim).

And min and max?

The other side of the LimitRange: besides filling in, it forbids. A container that asks for more than max or less than min is rejected. The max protects the cluster from the greedy Pod; the min protects developers from themselves (a ridiculous request produces Pods the scheduler places anywhere and that then suffocate).

Does this apply to the Pods that already existed?

No. Like Pod Security admission, the LimitRange acts at the door, on what is created from now on. The old Pods stay as they are.

Apply it and watch the perfect demonstration: create a Pod with no resources, the same command that failed a minute ago (prueba-limites means "limits test").

kubectl apply -f limitrange.yaml
kubectl run prueba-limites --image=ghcr.io/iximiuz/labs/nginx:alpine -n tienda

kubectl get pod prueba-limites -n tienda -o jsonpath='{.spec.containers[0].resources}'; echo

kubectl get pod prueba-limites -n tienda -o jsonpath='{.status.qosClass}'; echo

Now it gets in, and it gets in with resources you did not write: 100m and 64Mi of request, 200m and 128Mi of limit. The LimitRange injected them at admission, and with them the quota can now account for it.

And a consequence that closes the circle with the QoS lesson: that Pod, which in the default namespace would have been BestEffort, is created here as Burstable. Without anyone writing a single line of resources.

Step 4: The quota saying no

What remains is seeing the ceiling in action. Try to create a Pod that blows through the whole budget (tragon means "glutton"):

kubectl run tragon --image=ghcr.io/iximiuz/labs/nginx:alpine -n tienda \
  --overrides='{"spec":{"containers":[{"name":"tragon","image":"ghcr.io/iximiuz/labs/nginx:alpine","resources":{"requests":{"cpu":"10"}}}]}}'

Immediate error: exceeded quota. The Pod never even came to exist.

This is the difference worth engraving, because the three failures look alike and are not:

• Quota rejection: API error, instant. The object does not exist.
• LimitRange rejection (min or max): API error, instant. The object does not exist.
• Scheduler Pending: the object does exist, but nobody can find it a node. It is the case of the procesador Pod from that challenge.

Faced with a failure, the first question is always the same: did the object get created? The answer tells you whether the problem is in admission or in scheduling.

Summary

• The ResourceQuota is the namespace's hard budget: Pods, CPU, memory, objects, storage.
• The LimitRange fills in default requests and limits and imposes minimums and maximums per container.
• A quota without a LimitRange is half a solution: it forces everyone to declare resources and breaks every absent-minded manifest.
• Rejection at admission (the object does not exist) and scheduler Pending (the object exists with no node) are different failures.