Debug a Pod Stuck at 0/1 Ready Due to a Readiness Probe
Scenario
A new intern is learning how Kubernetes probes work. They created a Pod in the
cncf-54s63 namespace using this manifest:
/home/laborant/order-api.yaml
The Pod starts and keeps running, but never becomes Ready:
NAME READY STATUS RESTARTS AGE
order-api 0/1 Running 0 44s
The application and probe command are correct. The application writes
order-api ready to /tmp/ready as soon as it starts, and the probe command
succeeds when run manually inside the container. However, the readiness probe
fails on every attempt.
One probe parameter is missing. Kubernetes uses its default value, which does not give the probe command enough time to complete.
Task
Add the missing parameter with an appropriate value to
/home/laborant/order-api.yaml, then recreate the Pod from the updated
manifest so that order-api reports 1/1 Ready.
- Do not change the container
commandorargs. - Do not change the probe
exec.command. - Keep the readiness probe in place.
- Recreate the Pod using the updated manifest file.
Hint 1 - Identify how the probe is failing
There are two different ways a probe attempt can fail. Either the command runs to completion and returns a non-zero exit code, or Kubernetes stops the command before it has finished.
Read the Unhealthy events and work out which of the two is happening here:
kubectl describe pod order-api -n cncf-54s63
The distinction matters. The first case points at the application or the command. The second points at the probe's configuration.
Documentation
Hint 2 - Compare the timing values against the command
Higher up in the same describe output, inside the container details, one line summarises the probe's effective configuration:
Readiness: exec [<command>] delay=<n>s timeout=<n>s period=<n>s #success=<n> #failure=<n>
Open /home/laborant/order-api.yaml alongside it. Some of those values the
intern wrote. At least one they did not, and Kubernetes supplied a default.
Two of the timing settings matter here, and they mean different things:
- one sets the maximum duration of a single probe execution
- one sets the interval between probe executions
Measure how long the probe command actually takes:
kubectl exec order-api -n cncf-54s63 -- /bin/sh -c "time <the probe command>"
Then compare that duration against both settings and decide which one the command is exceeding.
Documentation
Hint 3 - How the probe parameters fit together
A readiness probe runs on a loop for the whole life of the container:
- Wait
initialDelaySecondsafter the container starts. - Run the probe command. If it has not finished within
timeoutSeconds, stop it and record a failure. - Wait
periodSeconds, then run it again. - After
successThresholdconsecutive passes, mark the container Ready. AfterfailureThresholdconsecutive failures, mark it not Ready.
| Parameter | What it controls | Default |
|---|---|---|
initialDelaySeconds | Delay after container start before the first attempt | 0 |
periodSeconds | Interval between attempts | 10 |
timeoutSeconds | Maximum duration of a single attempt | 1 |
successThreshold | Consecutive passes needed to become Ready | 1 |
failureThreshold | Consecutive failures before being marked not Ready | 3 |
Note that timeoutSeconds and periodSeconds are unrelated. One bounds how
long an attempt may run, the other decides how often attempts begin. A slow
command is constrained by the first, not the second.
Documentation
Hint 4 - The parameter, and how to apply it
The missing parameter is timeoutSeconds. It caps a single probe attempt and
defaults to 1 second when omitted. Any attempt running longer is stopped and
recorded as a failure, whatever it would have returned.
Set it to a whole number of seconds greater than the measured duration of the command. Anything from that point upward is acceptable, so pick a value that leaves some headroom rather than one that only just fits:
readinessProbe:
exec:
command: [...]
initialDelaySeconds: 2
periodSeconds: 3
timeoutSeconds: <value>
failureThreshold: 3
Probe fields are immutable on a running Pod, so re-applying the edited file on
its own will be rejected with Forbidden: pod updates may not change fields other than .... Delete the Pod first, then apply the updated manifest:
kubectl delete pod order-api -n cncf-54s63
kubectl apply -f /home/laborant/order-api.yaml
Documentation