Vertical Pod Autoscaler

The HPA answers the question "how many replicas do I need?". The Vertical Pod Autoscaler answers a very different one, and probably a more frequent one: "how big should each replica be?".

Because almost nobody knows how many millicores their application needs. You copy the resources from the microservice next door, round up just in case, and the cluster ends up full of Pods that reserve half a core to consume thirty millicores. Those inflated requests are not free: the scheduler believes them, reserves the space, and you end up paying for nodes to host air.

Two warnings before you start:

The VPA does not ship with Kubernetes. It is a component of the autoscaler project that has to be installed separately (the initialization of this lesson has already done it for you, with Helm).
This lesson uses the VPA in its most useful and least dangerous mode: recommendation only.

Work from the dev-machine tab. First confirm that the recommender is running:

kubectl get pods -n vpa

kubectl get crd verticalpodautoscalers.autoscaling.k8s.io

Step 1: A Deployment with inflated requests

Create holgazan.yaml (holgazan is Spanish for "slacker"): an nginx that does absolutely nothing and asks for half a core and 512Mi.

cat << 'EOF' > holgazan.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: holgazan
  namespace: tienda
spec:
  replicas: 2
  selector:
    matchLabels:
      app: holgazan
  template:
    metadata:
      labels:
        app: holgazan
    spec:
      containers:
      - name: nginx
        image: ghcr.io/iximiuz/labs/nginx:alpine
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: 500m
            memory: 512Mi
          limits:
            cpu: 1
            memory: 1Gi
EOF

kubectl apply -f holgazan.yaml
kubectl top pods -l app=holgazan

Compare the two figures: what it asks for (500m) and what it consumes (a few millicores). Multiply that difference by two hundred Pods and you have a lot of people's cloud bill.

Step 2: The VPA in recommendation mode

Create vpa.yaml:

cat << 'EOF' > vpa.yaml
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: holgazan-vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: holgazan
  updatePolicy:
    updateMode: "Off"
  resourcePolicy:
    containerPolicies:
    - containerName: nginx
      minAllowed:
        cpu: 10m
        memory: 32Mi
      maxAllowed:
        cpu: 1
        memory: 1Gi
EOF

The YAML, explained in questions and answers

Why apiVersion: autoscaling.k8s.io/v1 and not autoscaling/v2 like the HPA?

Because the VPA is not part of Kubernetes: its resources arrive as CRDs installed by the autoscaler project, and that is why they live in their own API group. It is the same extension mechanism you saw with the Gateway API.

What does targetRef do?

It points at the object whose Pods will be analyzed. Like the HPA's scaleTargetRef, it points at the controller (here a Deployment), not at the Pods.

What are the updateMode values, and which one is dangerous?

Four, and it helps to know them before touching production:

Off: it only calculates and publishes recommendations. It touches nothing. It is the one we use here, and it is where everyone should start.
Initial: it applies the recommendation only to new Pods, when they get created for some other reason. Prudent.
Recreate and Auto: they kill the Pods to recreate them with the corrected resources. Here is the VPA's historical fine print: since a Pod's resources were immutable, the only way to change them was to destroy the Pod. An autoscaler that restarts your Pods without warning is exactly as alarming as it sounds, and that is why the VPA has been used far less than it deserved.

And is that still the case?

It is changing: Kubernetes has been adding in-place resizing of a Pod's resources, without recreating it, and that is the natural future of the VPA in automatic mode. Meanwhile, the practical rule has not changed: in production, Off, and you apply the recommendations yourself, in your repository, after reviewing them.

What are minAllowed and maxAllowed for?

They are the guardrails, like the HPA's minReplicas and maxReplicas. They stop an absurd recommendation (from a metric gone haywire or a one-off spike) from leaving the Pod without resources or asking for the whole node.

Apply it and wait. The recommender needs to collect a few samples from metrics-server, so give it a few minutes:

kubectl apply -f vpa.yaml

kubectl get vpa holgazan-vpa

kubectl describe vpa holgazan-vpa

Step 3: Reading the verdict

When the Recommendation section of the describe shows up, read it carefully, because it has four figures and each one answers a different question:

kubectl describe vpa holgazan-vpa

target: what the VPA would set as the request right now. It is the figure you take to the YAML.
lowerBound: below this, the Pod would run short. If your current request is below it, you have a problem today.
upperBound: above this you are wasting for sure. Compare the 500m you asked for with this number.
uncappedTarget: what it would recommend without the minAllowed and maxAllowed guardrails. If it differs from target, your bounds are trimming the recommendation, and that is worth knowing.

Compare the target with your 500m and you will have, at a glance, the difference between what you thought you needed and what you need.

The conflict you need to know about

Putting an HPA and a VPA on the same Deployment and the same metric is a bad idea. The HPA sees high CPU and adds replicas; that lowers the average CPU per Pod; the VPA sees Pods that consume little and trims their requests; with lower requests, the same consumption becomes a higher utilization percentage; the HPA scales again. The two chase each other's tail.

The combinations that do work: HPA on CPU and VPA only in Off mode (recommending, while you decide), or HPA on a custom metric (requests per second, queue length) and VPA governing CPU and memory. Each one watching a different signal.

Summary

The VPA corrects the size of the Pod; the HPA, the number of Pods.
It does not come out of the box: it is installed as CRDs plus a recommender.
updateMode: Off is where you start and where almost everyone stays: the VPA as an advisor, not as a surgeon.
Four figures in the recommendation: target (the one you copy), lowerBound, upperBound and uncappedTarget.
Never an HPA and a VPA on the same metric of the same Deployment.

Course progress