Node affinity and selection

The scheduler does a good job by default: it looks at where the Pod fits, scores the nodes and picks one. But there are decisions it cannot make for you, because they depend on things it does not know. That this job needs SSD disk. That this cache must sit right next to the application that uses it. That the two replicas of the same service should not share a node, because then the node is a single point of failure.

There are four tools to influence that decision, and the usual mistake is to use the strongest one when the softest would have done.

The playground's nodes come already labeled. Look at them from the dev-machine tab:

kubectl get nodes --show-labels

node-01 has disktype=ssd and zona=a; node-02, disktype=hdd and zona=b. Notice also the labels Kubernetes puts on every node by itself, which you can use like any other: kubernetes.io/hostname, kubernetes.io/arch, kubernetes.io/os and, in a real cloud, topology.kubernetes.io/zone and /region.

Step 1: nodeSelector, the hammer

You already used it in the scheduling challenge, so just the reminder:

spec:
  nodeSelector:
    disktype: ssd

It is a strict equality filter and allows no nuance: either there is a node that matches, or the Pod stays in Pending forever. It is perfect when the rule is simple and non-negotiable, and it falls short as soon as you want to say "I'd prefer, but I don't require" or "either of these two values".

Step 2: nodeAffinity, the same hammer with nuances

The db of the tienda wants fast disk, but not at any price: if there is no node with SSD, it would rather start anyway than stay in Pending forever. That is exactly what nodeAffinity expresses. Create db.yaml:

cat << 'EOF' > db.yaml
apiVersion: v1
kind: Pod
metadata:
  name: db
  labels:
    app: db
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: disktype
            operator: In
            values:
            - ssd
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 50
        preference:
          matchExpressions:
          - key: zona
            operator: In
            values:
            - a
  containers:
  - name: app
    image: ghcr.io/iximiuz/labs/nginx:alpine
EOF

The YAML, explained in questions and answers

Are the fields really called that?

Yes, and their name is a treatise in itself. requiredDuringScheduling means "this is mandatory at scheduling time". IgnoredDuringExecution means "and once scheduled, I don't care": if tomorrow someone removes the label from the node, your Pod does not move. It is consistent with the whole Kubernetes philosophy: scheduling decisions are made once, when the Pod is placed.

What does this add over a nodeSelector?

Two things the nodeSelector cannot give you. First, operators: In, NotIn, Exists, DoesNotExist, Gt, Lt. You are no longer tied to equality. Second, and more important, the soft version.

What does the preferred part do with its weight?

It is a preference, not a requirement. The scheduler adds up the weights of the preferences each node satisfies and picks the one with the highest score, but if none satisfies them, it places the Pod anyway. It is the difference between "I want to be in zone A" and "don't start unless you are in zone A", and it is the right tool 80% of the time. A required that is not met leaves a Pod in Pending; a preferred that is not met leaves a Pod running in a suboptimal place. Think about which of the two you would rather have at three in the morning.

kubectl apply -f db.yaml
kubectl get pod db -o wide

On node-01, the only one with SSD.

Step 3: podAffinity, "I want to be near that one"

The previous two talk about nodes. These two talk about Pods, and it is a conceptual leap: the condition is no longer "what the node is like", but "who else lives on it".

Create cache.yaml: a cache that wants to be on the same node as the analytics (the db Pod), so that the network latency between them is zero.

cat << 'EOF' > cache.yaml
apiVersion: v1
kind: Pod
metadata:
  name: cache
  labels:
    app: cache
spec:
  affinity:
    podAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchLabels:
            app: db
        topologyKey: kubernetes.io/hostname
  containers:
  - name: app
    image: ghcr.io/iximiuz/labs/nginx:alpine
EOF

What on earth is topologyKey?

The key piece, and the one almost nobody gets the first time. It does not say "on the same node": it says "on the same value of this label". With kubernetes.io/hostname, two nodes are "the same place" only if they are literally the same node. Change the key to topology.kubernetes.io/zone and "the same place" comes to mean "the same availability zone", so the cache could go to any node in the analytics' zone. The same rule, two completely different ranges, depending on the label you choose as the unit of measure.

And who does the labelSelector point to?

To the reference Pods (the ones carrying app: db), not to the nodes. The scheduler looks at where those Pods are and places the new one accordingly.

kubectl apply -f cache.yaml
kubectl get pods -o wide -l 'app in (db,cache)'

Same node. And notice the side effect: the Pod cache ended up on node-01 without mentioning node-01 anywhere. It tied itself to another Pod, and that Pod dragged the decision along.

Step 4: podAntiAffinity, "don't put me with my clones"

The same idea, in the negative, and by far the most frequent use of the family: spreading the replicas of a service so that a node going down does not take them all with it. Create cache-ha.yaml, with a name and a label of its own so it does not get mixed up with the Pod cache from the previous step, which is still alive:

cat << 'EOF' > cache-ha.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cache-ha
spec:
  replicas: 2
  selector:
    matchLabels:
      app: cache-ha
  template:
    metadata:
      labels:
        app: cache-ha
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app: cache-ha
            topologyKey: kubernetes.io/hostname
      containers:
      - name: nginx
        image: ghcr.io/iximiuz/labs/nginx:alpine
EOF

Notice the elegant detail: the labelSelector points to its own Pods. The rule reads "don't put me where there is already one like me".

kubectl apply -f cache-ha.yaml
kubectl rollout status deployment/cache-ha --timeout=90s

kubectl get pods -o wide -l app=cache-ha

One Pod on each node.

And that label of its own is not a whim: had the Deployment reused app: cache, its rule would have counted the bare Pod from the previous step too, which occupies node-01. The first replica would have gone to node-02 and the second would have stayed in Pending forever, with not a single node free of clones. Which is exactly what happens next.

What if I ask for 5 replicas and only have 2 nodes? The 3 left over stay in Pending forever, because the rule is hard and there are no clone-free nodes. That is the price of required, and it is the reason many teams use the preferred version here, or go straight to topologySpreadConstraints (the one from the PDB lesson), which lets you say "spread as best you can, but don't leave me with Pods that never start".

Step 5: nodeName, the shortcut you should not use

There is a fifth mechanism, and it is the bluntest of all:

spec:
  nodeName: node-01

What exactly does it do? It writes directly the field the scheduler would have filled in. In other words: the Pod arrives already scheduled, and the scheduler does not even look at it. No resource checks, no taints, no affinities. If the node does not exist or is full, the Pod is left hanging there and nobody is going to rescue it.

Remember the first lesson of the course: a Pod with no nodeName is an orphaned Pod waiting for the scheduler. nodeName is filling in that box by hand and skipping the queue.

Its legitimate uses are few: debugging, and the static Pods of the control plane (which is precisely why they can start before a scheduler exists). In an application, never.

Comparative summary

And remember the other half of the vocabulary, the one from the previous challenge: taints and tolerations do not choose a destination, they only open doors. A taint repels; a toleration stops you from being repelled. To go somewhere you need one of the tools in this lesson.