Devices and Dynamic Resource Allocation

Everything you have handed out so far was CPU and memory: divisible, homogeneous resources, where it does not matter which core those 500 millicores come from. A device does not work like that. A GPU has a model, memory of its own and a position on the bus, and it is assigned whole or in slices its driver decides, not you. That is why you don't ask for it with requests: you claim it.

The recomendador is the piece of the tienda that exists for this, and it is the only one that never gets to start in the book. In this playground it is not going to start either: there is no GPU on these machines. But that is not the end of the lesson: it is the lesson. You are going to apply the claim and read the four messages the cluster gives you, one for each piece that is missing, until you reach the only point where real hardware is needed.

Work from the dev-machine tab.

Step 1: The types are already there

Before applying anything, check one thing:

kubectl api-resources --api-group=resource.k8s.io

Four types: DeviceClass, ResourceClaim, ResourceClaimTemplate and ResourceSlice. Nobody has installed anything on this cluster. Dynamic Resource Allocation is core API, not a CRD the vendor brings along: the vendor installs the driver, but the types were already in place. It is the same cast of roles as in the Storage chapter (DeviceClass plays the StorageClass, ResourceClaim the PVC, ResourceSlice the PersistentVolume) with a driver behind it, as in CSI.

Now create recomendador.yaml:

cat << 'EOF' > recomendador.yaml
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: gpu-24gb
spec:
  spec:
    devices:
      requests:
      - name: gpu
        exactly:
          deviceClassName: gpu-inferencia
          allocationMode: ExactCount
          count: 1
          selectors:
          - cel:
              expression: |-
                device.capacity["gpu.example.com"].memory
                  .compareTo(quantity("24Gi")) >= 0
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: recomendador
spec:
  replicas: 1
  selector:
    matchLabels:
      app: recomendador
  template:
    metadata:
      labels:
        app: recomendador
    spec:
      resourceClaims:
      - name: gpu
        resourceClaimTemplateName: gpu-24gb
      containers:
      - name: recomendador
        image: ghcr.io/iximiuz/labs/nginx:alpine
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          claims:
          - name: gpu
EOF

What does that claim say? It does not say "a GPU". It says "a device of the class gpu-inferencia, exactly one, and with at least 24 GiB of memory". That last condition is the CEL expression, and it is DRA's reason to exist: with the classic device plugin you can only ask for nvidia.com/gpu: 1, which means "one, whichever". Here you can choose.

And why a template rather than a plain ResourceClaim? Because a ResourceClaim is a single object: if the Deployment had three replicas, they would share it, and that is almost never what you want with a device. With the template, Kubernetes generates one claim per Pod.

Apply it and look at both things at once:

kubectl apply -f recomendador.yaml

kubectl get pods -l app=recomendador

kubectl get resourceclaim

The Pod is in Pending, as you expected. But the ResourceClaim exists, with a name derived from the Pod's, and it is pending too. Check whose it is:

kubectl get resourceclaim -o jsonpath='{.items[0].metadata.ownerReferences[0].kind}'

Pod. That claim was created with the Pod and will disappear with it. There is the difference between template and claim, in one command instead of a paragraph.

Step 2: The first message

A Pod in Pending says nothing on its own. The event does:

kubectl describe pod -l app=recomendador | tail -5

0/3 nodes are available: request gpu: device class gpu-inferencia
does not exist.

The class does not exist. And it does not exist because the class is not written by the application: whoever operates the cluster creates it, just like the StorageClass. Put on that hat for a moment:

cat << 'EOF' > deviceclass.yaml
apiVersion: resource.k8s.io/v1
kind: DeviceClass
metadata:
  name: gpu-inferencia
spec:
  selectors:
  - cel:
      expression: device.driver == "gpu.example.com"
EOF

Notice that the DeviceClass has no namespace (it is cluster-scoped) and that its selector does not say how many GPUs there are or how much memory they have: it says which devices belong to this class. "How much I want" is the claim's business; "what kind there is" is the class's.

kubectl apply -f deviceclass.yaml

And now don't touch the Pod. There is no need to delete it or restart anything: the scheduler retries pending Pods on its own when the state of the cluster changes. Wait a few seconds and look again:

kubectl describe pod -l app=recomendador | tail -5

0/3 nodes are available: 3 cannot allocate all claims.

Different message, different piece. The class is no longer missing: the device is. Nobody has published any GPU on any node, so there is nothing to allocate. That is what a DRA driver does, and this cluster has none.

Step 3: Publish a GPU that does not exist

A ResourceSlice is what the driver publishes on each node with what it finds there. Nobody writes one by hand... except today, to see the mechanism from the inside. Publish two devices with different memory sizes, which is exactly the case the selector has to resolve:

cat << 'EOF' > resourceslice.yaml
apiVersion: resource.k8s.io/v1
kind: ResourceSlice
metadata:
  name: nodo-simulado-gpu
spec:
  driver: gpu.example.com
  nodeName: node-01
  pool:
    name: node-01
    generation: 1
    resourceSliceCount: 1
  devices:
  - name: gpu-0
    capacity:
      memory:
        value: 24Gi
  - name: gpu-1
    capacity:
      memory:
        value: 8Gi
EOF

kubectl apply -f resourceslice.yaml

kubectl get resourceclaim

kubectl get pods -l app=recomendador -o wide

The claim goes to allocated,reserved and the Pod is scheduled on node-01. Now the question that matters: which of the two did it get?

kubectl get resourceclaim \
  -o jsonpath='{range .items[*]}{.status.allocation.devices.results[0].device}{end}'

gpu-0. The 8 GiB one was never even considered, because your CEL expression rules it out. The kube-scheduler has cross-checked two things at once (which nodes have devices that fit, and which have room) and chose node and device in the same decision. That is what a classic device plugin cannot do: it only knows how to count.

Step 4: Where the simulation ends

The Pod has a node, has a device allocated... and still does not start. Look at the last event:

kubectl describe pod -l app=recomendador | tail -3

Failed to prepare dynamic resources: DRA driver gpu.example.com
is not registered

And here the trick ends, in the most honest way possible. You have described a device, but describing it does not make it exist: what is missing is the kubelet plugin that prepares that GPU on the node and hands it to the container. That can no longer be simulated with YAML.

Now look at the four events in a row, in order:

kubectl get events --field-selector involvedObject.kind=Pod --sort-by=.lastTimestamp

Four rows, trimmed here to what matters (the REASON and the end of the MESSAGE):

Warning  FailedScheduling               ... device class gpu-inferencia does not exist
Warning  FailedScheduling               ... 3 cannot allocate all claims
Normal   Scheduled                      Successfully assigned default/recomendador-... to node-01
Warning  FailedPrepareDynamicResources  ... DRA driver gpu.example.com is not registered

Four lines and four pieces, each naming exactly what it was missing: the class, the device, the node (that one it did get) and the driver. No Pod will ever tell you better how DRA works.

Both flags are needed. --sort-by=.lastTimestamp because kubectl get events does not guarantee chronological order and here the order is precisely what you read. And the --field-selector because, without it, the Deployment's and ReplicaSet's events show up in between ("scaled up", "created pod") and tell you nothing about DRA.