Probes, the state of the Pod

Kubernetes knows nothing about your application. It knows a process is alive because the kernel says so, and that is where its intuition ends: a live process can be stuck, still starting, disconnected from the database or completely useless. The probes are the only channel through which the application tells the cluster how it is really doing.

There are three of them, and mixing them up is one of the best ways to take a service down in production:

• readiness: can I receive traffic? If it fails, the Pod leaves the Service's endpoints. It is not killed.
• liveness: am I still working? If it fails, the kubelet kills the container and restarts it.
• startup: have I finished starting? Until it succeeds, the other two are not even evaluated.

Work from the dev-machine tab.

Step 1: A Deployment with real probes

Create the file probes.yaml. It brings a Deployment and a Service together, because the point of readiness only shows with a Service in front:

cat << 'EOF' > probes.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 2
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
      - name: nginx
        image: ghcr.io/iximiuz/labs/nginx:alpine
        command: ["sh", "-c"]
        args:
        - |
          touch /tmp/listo
          exec nginx -g 'daemon off;'
        ports:
        - containerPort: 80
        readinessProbe:
          exec:
            command: ["cat", "/tmp/listo"]
          periodSeconds: 3
          failureThreshold: 2
        livenessProbe:
          httpGet:
            path: /
            port: 80
          periodSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
  name: api
spec:
  selector:
    app: api
  ports:
  - port: 80
    targetPort: 80
EOF

The YAML, explained in questions and answers

Why is the readiness probe exec and the liveness probe httpGet?

Because we want to be able to break one without breaking the other. The readiness probe checks a file (/tmp/listo, "listo" meaning ready) that you are going to delete by hand; the liveness probe checks that nginx keeps serving HTTP, which it will. That way we isolate the effect of each one. In a real application both are usually HTTP, but pointing at different endpoints: /readyz looks at the dependencies (database, caches) and /healthz only checks that the process is not stuck.

What do periodSeconds and failureThreshold mean?

How often the check runs and how many consecutive failures it takes to declare it failed. With periodSeconds: 3 and failureThreshold: 2, a Pod takes about 6 seconds to leave the endpoints. The other timers in the family are initialDelaySeconds (wait before the first attempt), timeoutSeconds (how long to tolerate a slow answer, 1 second by default, which falls short more often than it seems) and successThreshold (how many consecutive successes it takes to declare it good again).

What happens if I declare no probe at all?

Kubernetes uses the only criterion it has: the container is alive if its main process has not exited. That is the default behavior of everything you have deployed so far, and it explains why a Pod can be Running and READY 1/1 while the application returns 500 errors to everyone.

Do the three probe types support the same mechanisms?

Yes: httpGet (200 to 399 is success), tcpSocket (it is enough that the port accepts the connection) and exec (exit code 0). There is also grpc for services that implement the gRPC health checking protocol.

Apply it and check the starting picture:

kubectl apply -f probes.yaml

kubectl get pods -l app=api

kubectl get endpointslice -l kubernetes.io/service-name=api

Two Pods ready, two endpoints in the Service.

Step 2: Break the readiness of a single Pod

Here is the experiment that gives this lesson its point. Pick one of the two Pods and delete the file its readiness probe checks:

kubectl get pods -l app=api

To avoid typing the Pod name in the next commands, pick a Pod and store its name in the POD variable: export POD=<one-of-the-two-pods>

kubectl exec $POD -- rm /tmp/listo

And now watch for a few seconds:

kubectl get pods -l app=api --watch

The Pod goes to READY 0/1, but notice the RESTARTS column: still at zero. The container is alive, nginx answers perfectly, it has simply said "don't send me traffic right now". Confirm it where it matters:

kubectl get pod $POD -o jsonpath='{.status.conditions}' | jq .

Check the state of the EndpointSlice:

kubectl get endpointslice -l kubernetes.io/service-name=api -o go-template='
{{- printf "%-35s | %-15s | %-12s | %-6s | %-7s | %-11s\n" "targetRef.name" "ip address" "port info" "ready" "serving" "terminating" -}}
{{- range .items -}}
  {{- range .endpoints -}}
    {{- printf "%-35s | %-15s | %-12s | %-6v | %-7v | %-11v\n" .targetRef.name (index .addresses 0) (printf "%d/%s" (index (index $.items 0).ports 0).port (index (index $.items 0).ports 0).protocol) .conditions.ready .conditions.serving .conditions.terminating -}}
  {{- end -}}
{{- end -}}
'

kubectl supports several output formats. Use whichever suits you best in each situation: https://kubernetes.io/docs/reference/kubectl/#output-options

A single endpoint. The Service has stopped sending requests to the Pod that failed its readiness probe without killing it, and the other Pod absorbs all the traffic. That is the promise of readiness: stepping out of the service is reversible and does not cost a restart.

Put it back into service to see the way back (recovery is automatic too):

kubectl exec $POD -- touch /tmp/listo
kubectl get endpointslice -l kubernetes.io/service-name=api

Step 3: The startup probe and slow processes

Now the other classic failure. Imagine an application with a long startup (a JVM, a migration, a cache being built). Simulate it with this Pod, which takes 40 seconds to be ready.

Create lento.yaml (lento means "slow") without a startup probe, exactly like this:

cat << 'EOF' > lento.yaml
apiVersion: v1
kind: Pod
metadata:
  name: lento
spec:
  containers:
  - name: app
    image: ghcr.io/iximiuz/labs/nginx:alpine
    command: ["sh", "-c"]
    args:
    - |
      sleep 40
      touch /tmp/listo
      exec nginx -g 'daemon off;'
    livenessProbe:
      exec:
        command: ["cat", "/tmp/listo"]
      periodSeconds: 5
      failureThreshold: 1
EOF

Apply it and watch the disaster (be patient and wait until you see the Pod change state):

kubectl apply -f lento.yaml
kubectl get pod lento --watch

CrashLoopBackOff. The liveness probe starts checking after a few seconds, the file does not exist yet, and the kubelet kills the container before the application gets to start. The Pod will never get out of that loop: every attempt starts again from zero.

The temptation is to raise initialDelaySeconds to 60. It is a bad solution: if you guess too low you keep killing the startup, and if you guess too high you take a minute to detect a real hang. The right answer is the startup probe.

What exactly does a startup probe do?

It disables the other two until it succeeds for the first time. You give it a generous budget of attempts (failureThreshold times periodSeconds) and, as soon as the process starts, it switches off for good and the liveness probe takes over with aggressive timers. Slow startup and fast hang detection, without having to choose.

Fix the manifest by adding the startup probe, delete the Pod and apply it again (probes, like almost everything in the spec, are immutable):

    startupProbe:
      exec:
        command: ["cat", "/tmp/listo"]
      periodSeconds: 5
      failureThreshold: 20

Twenty attempts every 5 seconds are 100 seconds of margin for a 40-second startup. The liveness probe stays as it is, with its failureThreshold: 1, because once the process has started we do want to kill it fast if it hangs.

kubectl delete pod lento
kubectl apply -f lento.yaml
kubectl get pod lento --watch

Forty seconds of patience and the Pod reaches Running and READY 1/1, with zero restarts.

Summary

• readiness: takes you out of the Service. Does not kill. It is the one that protects your users.
• liveness: restarts you. It must only look inward, never at an external dependency.
• startup: silences the other two during startup. It is the right answer to the CrashLoopBackOff of slow processes.
• A Running Pod without probes only guarantees that a process exists, not that it is good for anything.

The next lesson brings the other half of the Pod's contract with the cluster: resources, limits and QoS classes. And right after it, a challenge where one of these probes is pointed at the wrong place.