This challenge was created by a community author. Community content is reviewed by the iximiuz Labs team on a best effort basis.

Challenge

Learner activity No attempts yet

Troubleshoot HPA Pod Scaling Blocked by ResourceQuota

by Omkar Shelke · on

Kubernetes

A ResourceQuota is limiting pod count to 1 in the finops namespace, preventing the HPA from scaling the billing-api deployment to its minimum of 3 pods. Identify the conflict and update the ResourceQuota to allow the HPA to function correctly.

#cka #ckad #hpa #resourcequota #autoscaling

Scenario

The billing-api service is running in the finops namespace with a Horizontal Pod Autoscaler (HPA) configured. However, the deployment is not reaching the HPA minimum replica count and remains at 1 pod.

Investigate the ResourceQuota configuration, identify what is causing the scaling issue according to the HPA configuration, and fix it.

Task

The billing-api deployment has an HPA configured with:

minReplicas: 3
maxReplicas: 10

Inspect the ResourceQuota in the namespace, determine what is causing autoscaling issues according to the HPA configuration, and make the necessary changes so the deployment can scale according to the configured HPA range.

Important

Wait approximately 1 minute for the environment setup and HPA metrics collection to complete.

Do not delete or modify the HPA or Deployment.

Hint

When pods are not scaling as expected, check not just the HPA but also any namespace-level policies that could be restricting pod creation.

# check HPA status and events
kubectl describe hpa billing-api-hpa -n finops

# list all resources in the namespace
kubectl get all -n finops

# check namespace-level restrictions
kubectl get resourcequota -n finops
kubectl describe resourcequota -n finops

Test Cases

Understanding the Solution

This challenge demonstrates a common conflict between ResourceQuota and HorizontalPodAutoscaler (HPA) — the quota limits pod count to 1, but the HPA requires a minimum of 3 pods. Kubernetes enforces the quota at the namespace level, so even though the HPA wants to create more pods, the API server rejects the requests.

Why the scaling is blocked

HPA wants:   minReplicas=3, maxReplicas=10
             CPU utilization is 167% — well above 50% threshold
             → HPA tries to scale up to 4 pods

ResourceQuota says:  pods: "1"
             → API server rejects every new pod creation

Result:  Only 1 pod runs. HPA keeps retrying and failing.

The events confirm this:

Error creating: pods "billing-api-xxx" is forbidden:
exceeded quota: finops-quota, requested: pods=1, used: pods=1, limited: pods=1

The fix is to update finops-quota so the pod limit is at least equal to the HPA maxReplicas (10) — this allows the HPA to scale freely within its configured range.

Step 1 : Identify the problem

Check the HPA status:

kubectl get hpa billing-api-hpa -n finops

NAME              REFERENCE                TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
billing-api-hpa   Deployment/billing-api   cpu: 167%/50%   3         10        4          5m

CPU is at 167% — far above the 50% target. The HPA wants to scale up but only 1 replica is running.

Check the events to find the root cause:

kubectl get events -n finops | grep -i warning

Warning   FailedCreate   replicaset/billing-api-8d77c8cf9
Error creating: pods "billing-api-xxx" is forbidden:
exceeded quota: finops-quota, requested: pods=1, used: pods=1, limited: pods=1

The FailedCreate events clearly show every new pod is being rejected by the quota.

Step 2 : Inspect the ResourceQuota

kubectl describe resourcequota finops-quota -n finops

Name:                   finops-quota
Namespace:              finops
Resource                Used   Hard
--------                ----   ----
configmaps              1      10
limits.cpu              500m   8
limits.memory           128Mi  8Gi
persistentvolumeclaims  0      5
pods                    1      1      ← this is the problem
requests.cpu            300m   4
requests.memory         64Mi   4Gi
secrets                 0      10
services                0      10

pods: 1 — the namespace is capped at 1 pod total. The HPA needs at least 3 (minReplicas) and up to 10 (maxReplicas). The quota must be updated to allow the full HPA range.

Step 3 : Update the ResourceQuota

The pod limit must be set to 10 — matching the HPA maxReplicas so the HPA can scale freely within its configured range:

kubectl patch resourcequota finops-quota -n finops \
  --type=merge \
  -p '{"spec": {"hard": {"pods": "10"}}}'

resourcequota/finops-quota patched

Alternatively using kubectl edit:

kubectl edit resourcequota finops-quota -n finops

Change:

spec:
  hard:
    pods: "1"    # ← change this

To:

spec:
  hard:
    pods: "10"   # ← matches HPA maxReplicas

Note

Setting pods equal to maxReplicas (10) is the correct value — it allows the HPA to scale up to its maximum without restriction. Setting it to any value between 3 and 9 would still block scaling before the HPA max is reached.

Step 4 : Verify the ResourceQuota is updated

kubectl describe resourcequota finops-quota -n finops

Name:                   finops-quota
Namespace:              finops
Resource                Used   Hard
--------                ----   ----
configmaps              1      10
limits.cpu              500m   8
limits.memory           128Mi  8Gi
persistentvolumeclaims  0      5
pods                    1      10     ← updated
requests.cpu            300m   4
requests.memory         64Mi   4Gi
secrets                 0      10
services                0      10

The hard limit for pods is now 10.

Step 5 : Watch the HPA scale the deployment

Now that the quota allows more pods, the HPA immediately starts creating the pods it was blocked from creating:

kubectl get pods -n finops -w

NAME                          READY   STATUS              RESTARTS   AGE
billing-api-8d77c8cf9-dszlf   1/1     Running             0          7m
billing-api-8d77c8cf9-x8zfj   0/1     ContainerCreating   0          3s
billing-api-8d77c8cf9-pnkjv   0/1     ContainerCreating   0          3s
billing-api-8d77c8cf9-sls9q   0/1     ContainerCreating   0          4s

Check the HPA:

kubectl get hpa billing-api-hpa -n finops

NAME              REFERENCE                TARGETS        MINPODS   MAXPODS   REPLICAS   AGE
billing-api-hpa   Deployment/billing-api   cpu: 60%/50%   3         10        4          8m

The HPA is now scaling the deployment. Replicas are increasing as CPU load is distributed across more pods.

Step 6 : Confirm at least 3 pods are running

kubectl get deployment billing-api -n finops

NAME          READY   UP-TO-DATE   AVAILABLE   AGE
billing-api   3/3     3            3           9m

kubectl get pods -n finops

NAME                          READY   STATUS    RESTARTS   AGE
billing-api-8d77c8cf9-dszlf   1/1     Running   0          9m
billing-api-8d77c8cf9-x8zfj   1/1     Running   0          1m
billing-api-8d77c8cf9-pnkjv   1/1     Running   0          1m

At least 3 pods are running — the HPA minimum is now satisfied.