Lesson  in  Kubernetes 101

DaemonSet

One Pod on every node, without writing a replica count: the object used to deploy log agents, metrics and network plugins.

A Deployment answers the question "how many copies do I want?". There is a family of workloads for which that question makes no sense: the logs agent, the metrics exporter, the network plugin, the storage driver. Of all of them you want exactly one per node, no more and no less, and you want it to appear on its own when a new node joins the cluster.

That is the job of the DaemonSet. You have already run into one without knowing it: kube-proxy, from the lesson The cluster's components.

Work from the dev-machine tab.

Step 1: The DaemonSet

A logs agent is useless if it cannot read the logs, and the containers' logs live on the node's disk. So this DaemonSet mounts a directory of the host machine. Create daemonset.yaml:

cat << 'EOF' > daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: agent
  labels:
    app: agent
spec:
  selector:
    matchLabels:
      app: agent
  template:
    metadata:
      labels:
        app: agent
    spec:
      tolerations:
      - operator: Exists
      volumes:
      - name: logs-del-nodo
        hostPath:
          path: /var/log
      containers:
      - name: agent
        image: ghcr.io/iximiuz/labs/nginx:alpine
        command: ["sh", "-c", "while true; do sleep 3600; done"]
        volumeMounts:
        - name: logs-del-nodo
          mountPath: /var/log/host
          readOnly: true
        resources:
          requests:
            cpu: 10m
            memory: 32Mi
EOF

The YAML, explained in questions and answers

Where is the replicas field?

It does not exist, and that absence is the whole lesson. You do not decide the number of Pods: the number of nodes does. Add a node to the cluster and a Pod will appear; remove a node and it will disappear. That is why kubectl scale does not work on a DaemonSet.

Then who picks the node for each Pod?

The DaemonSet controller creates each Pod with a nodeAffinity that ties it to a specific node, and the scheduler merely confirms the decision. This matters when you debug: a DaemonSet Pod in Pending is not waiting for room to appear on any node, it is waiting to fit on its node.

Why that toleration with operator: Exists?

Because it tolerates any taint, and that is exactly what an infrastructure agent wants: to run also on the reserved or cordoned nodes, including the control plane. Remember the scheduling challenge: a taint rejects everyone who does not tolerate it, and a logs agent that does not watch the control plane is a logs agent with a blind spot. It is the usual pattern in system DaemonSets, although it helps to use it deliberately and not out of habit.

How is a DaemonSet updated?

With updateStrategy, which by default is RollingUpdate with maxUnavailable: 1: the Pods are replaced node by node. The alternative, OnDelete, touches nothing until you delete the Pods by hand, and it is used for agents so delicate that the replacement must be manual.

Why put the agent in another Namespace, instead of relaxing the policy of tienda?

Because the exception is granted to one piece, not to a neighborhood. If you lower the level of tienda so the agent fits, you also lower it for the frontend, the backend and the database, which do not need it and which are precisely the ones exposing traffic to the outside. Putting the privileged piece in its own Namespace keeps the exception contained and visible: whoever looks at plataforma knows that in there are permissions that do not exist in the rest of the cluster.

And if I only want the agent on some nodes?

You combine it with what you already know: a nodeSelector or a nodeAffinity in the template. It is the usual way to deploy, for example, a GPU driver only on the nodes that have one.

Step 2: Where it can NOT live

Apply it where you are, which is the Namespace tienda:

kubectl apply -f daemonset.yaml

kubectl get daemonset agent

kubectl get pods -l app=agent

The DaemonSet gets created, but not a single Pod appears. Ask why:

kubectl get events --field-selector involvedObject.name=agent --sort-by=.lastTimestamp

There it is: violates PodSecurity "baseline:latest": hostPath volumes. The Namespace tienda carries a label that enforces a minimum security level, and mounting a directory of the node does not meet it. The object exists, the controller tries, and the API rejects every Pod it creates.

And why does the error show up on the DaemonSet and not when applying the YAML?

Because the one violating the policy is not the DaemonSet, it is the Pods, and those are created by the controller afterward. It is the pattern of every controller: the parent object is admitted and the children are rejected one by one. That is why you have to look at the controller's events and not only at the output of the apply.

That mechanism is Pod Security Admission and it has its own lesson later on. For now keep the consequence: the agent does not fit in tienda.

Step 3: Where it can

Delete it and take it to plataforma, the Namespace the infrastructure team reserves for the pieces that need to touch the machine:

kubectl delete -f daemonset.yaml
kubectl apply -f daemonset.yaml -n plataforma

kubectl get daemonset agent -n plataforma

kubectl get pods -l app=agent -n plataforma -o wide

One Pod per node, each on its own. Notice the DaemonSet's columns: DESIRED, CURRENT, READY and NODE SELECTOR tell the same story from the controller's point of view.

Try to scale it, so the API explains it better than I can:

kubectl scale daemonset agent -n plataforma --replicas=1

Summary

  • A DaemonSet covers nodes, not replicas: the replicas field does not exist and scaling it makes no sense.
  • An agent that mounts hostPath does not fit in a Namespace with baseline: that is why it lives in plataforma and not in tienda. The exception is confined to one piece instead of being relaxed for the whole neighborhood.
  • The controller ties each Pod to its node, so a DaemonSet Pod in Pending is waiting for room on that node, not on any node.
  • Tolerations open the door of the reserved nodes for it; nodeSelector limits it to the nodes you want.
  • It is updated node by node with updateStrategy: RollingUpdate, or by hand with OnDelete.

In the next lesson, the other way of breaking the mold: several containers inside the same Pod.

Previous lesson
StatefulSet
Next lesson
Init containers and sidecars
Start lesson