This challenge was created by a community author. Community content is reviewed by the iximiuz Labs team on a best effort basis.

Challenge

Learner activity No attempts yet

Debug a Restarting Pod Using Previous Container Logs

by Omkar Shelke · on

Kubernetes Linux

A container is crashing because its command references a file that does not exist. Capture the previous container logs, diagnose the issue, fix the command, and verify the pod runs correctly.

#ckad #cka #debugging #logs

Scenario

A Pod named log-processor is running in the workloads namespace. The container keeps restarting because its command fails.

Because restartPolicy: Always is set, the container keeps restarting, giving you access to the previous container logs before investigating and fixing the issue.

Task

1. Save the previous container crash logs to /home/laborant/previous-logs.txt.

2. Read the logs to understand why the container is crashing.

3. Fix the container args based on what you find in the logs so the Pod stays Running.

Important

Important: Check the logs carefully to identify the exact failure. Do not remove, modify the existing args commands, but you may add the missing command required to fix the error before the failing command executes.

Hint 1 — Capture previous logs

When a container keeps restarting, the --previous flag lets you read the logs from the last terminated instance before it was restarted. Without it, you only see logs from the currently running container.

kubectl logs <pod> -n <namespace> -c <container> --previous

Save the output to a file so you can refer back to the error message while investigating.

kubectl logs

Hint 2 — Understand and resolve the crash

Read the error message carefully — it tells you exactly what failed and why. The error stat: can't stat '/logs/init.d/sas.log' means the file /logs/init.d/sas.log does not exist when the command runs. The fix is to ensure the required directory and log file are created before the failing step executes. Once you understand the error, update the args field of the container using:

kubectl edit pod log-processor -n workloads

Keep all existing steps intact — only add what is missing before the failing command.

Debugging Pods

Test Cases

Understanding the Solution

The container runs a shell script via args that executes steps in sequence. The original args tries to run stat /logs/init.d/sas.log but neither the directory /logs/init.d/ nor the file sas.log exist inside the container filesystem — so stat fails and the container exits with a non-zero code.

Because restartPolicy: Always is set, Kubernetes immediately restarts the container. The pod shows Running at all times — but it keeps cycling through the same steps, hitting the same error, and restarting. Each restart increments the RESTARTS counter.

Container starts
      ↓
echo "=== Container startup initiated... ==="
      ↓
sleep 15
      ↓
stat /logs/init.d/sas.log   ← fails — path does not exist
      ↓
Container exits with error
      ↓
Kubernetes restarts immediately (restartPolicy: Always)
      ↓
Pod shows Running again — RESTARTS counter goes up
      ↓
Loop repeats

This is why kubectl logs --previous is critical — the current running instance only shows the startup message during its sleep 15. The crash output lives in the previous terminated instance.

The fix is to add mkdir -p /logs/init.d && touch /logs/init.d/sas.log before stat so the path exists when stat runs.

Step 1 : Check the pod status

kubectl get pod log-processor -n workloads

NAME            READY   STATUS    RESTARTS      AGE
log-processor   1/1     Running   1 (13s ago)   45s

The pod shows Running — but RESTARTS: 1 tells you it has already crashed and been restarted once. This is not a healthy running pod — it will crash again after sleep 15 completes.

Step 2 : Capture the previous container logs

Check the current logs first — this shows what the currently running instance is doing:

laborant@dev-machine:~$ k get po -n workloads 
NAME            READY   STATUS    RESTARTS      AGE
log-processor   1/1     Running   2 (13s ago)   99s

kubectl logs log-processor -n workloads -c processor

=== Container startup initiated... ===

Note

The Pod is currently in the Running state because the container is still executing the sleep 15 command, so the failure has not occurred yet. Once sleep 15 completes, the stat command will fail, causing the container to crash.

Kubernetes will then restart the container according to the restart policy. After several consecutive failures, the kubelet applies exponential backoff by increasing the delay between restarts — eventually the Pod enters CrashLoopBackOff, where Kubernetes continues retrying but with progressively longer wait times between each restart.

Only the startup message — the container is still in its sleep 15 phase. The error has not happened yet in this instance. This is why you need --previous.

Use --previous to get logs from the last terminated container instance where the crash actually occurred:

kubectl logs log-processor -n workloads -c processor --previous

=== Container startup initiated... ===
stat: can't stat '/logs/init.d/sas.log': No such file or directory

Save the previous logs to the required file before making any changes:

kubectl logs log-processor -n workloads -c processor --previous \
  > /home/laborant/previous-logs.txt

Note

The pod shows Running because restartPolicy: Always restarts it immediately after every crash. The current instance is always in the early sleep 15 phase when you check — so current logs only show the startup message. The actual error only exists in the previous terminated instance, which is why --previous is the only way to see it.

Verify:

cat /home/laborant/previous-logs.txt

=== Container startup initiated... ===
stat: can't stat '/logs/init.d/sas.log': No such file or directory

Caution

kubectl logs -n workloads pod/log-processor --previous

Error:
unable to retrieve container logs for containerd://37d08d3601a561c217f4c944afc905cbbd5c6da3ba65af4acee860d01e08d395

The above error occurs because Kubernetes is unable to access logs from the previous container instance. The --previous flag works only when the container has restarted and the old logs are still available on the node. If the Pod was recreated, deleted, evicted, or containerd cleaned up the old logs, Kubernetes cannot retrieve them anymore.

Step 3 : Diagnose the issue from the logs

The error message is exact:

stat: can't stat '/logs/init.d/sas.log': No such file or directory

Look at the current args to understand the command sequence:

kubectl get pod log-processor -n workloads -o yaml | \
  yq '.spec.containers[] | select(.name == "processor") | .args'

- |
  echo "=== Container startup initiated... ==="
  sleep 15
  stat /logs/init.d/sas.log && sleep 3600

The problem is clear — stat runs on /logs/init.d/sas.log but the directory /logs/init.d/ is never created. Containers start with a minimal filesystem — paths like /logs/init.d/ do not exist unless explicitly created. The fix is to create the directory and file before stat runs.

Step 4 : Edit the pod and fix the args

kubectl edit pod log-processor -n workloads

Find the args section and update it:

Before:

args:
- |
  echo "=== Container startup initiated... ==="
  sleep 15
  stat /logs/init.d/sas.log && sleep 3600

After:

args:
- |
  echo "=== Container startup initiated... ==="
  sleep 15
  mkdir -p /logs/init.d && touch /logs/init.d/sas.log
  stat /logs/init.d/sas.log && sleep 3600

Save and exit. Because args is immutable on a running pod, kubectl edit will prompt you to force replace:

kubectl replace -f /tmp/kubectl-edit-<id>.yaml --force

pod "log-processor" deleted from workloads namespace
pod/log-processor replaced

Note

kubectl replace --force deletes and recreates the pod. This is expected for immutable pod fields. The pod name and namespace stay the same — this is why saving the previous logs in Step 2 before making any changes is important. After replace --force, the old previous logs are gone.

Step 5 : Verify the pod is Running with no restarts

kubectl get pod log-processor -n workloads

NAME            READY   STATUS    RESTARTS   AGE
log-processor   1/1     Running   0          25s

RESTARTS: 0 confirms the container is no longer crashing — the fix worked.

Step 6 : Check the current pod logs

Wait for sleep 15 to complete, then check the logs:

kubectl logs log-processor -n workloads -c processor

=== Container startup initiated... ===
  File: /logs/init.d/sas.log
  Size: 0               Blocks: 0          IO Block: 4096   regular empty file
Device: 2fh/47d Inode: 259753      Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2026-05-23 07:55:51.559473377 +0000
Modify: 2026-05-23 07:55:51.559473377 +0000
Change: 2026-05-23 07:55:51.559473377 +0000

stat now succeeds — the file metadata is printed and the container continues to sleep 3600.

Step 7 : Verify the file exists inside the container

kubectl exec log-processor -n workloads -c processor -- \
  stat /logs/init.d/sas.log

=== Container startup initiated... ===
  File: /logs/init.d/sas.log
  Size: 0               Blocks: 0          IO Block: 4096   regular empty file
Device: 2fh/47d Inode: 259753      Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2026-05-23 07:55:51.559473377 +0000
Modify: 2026-05-23 07:55:51.559473377 +0000
Change: 2026-05-23 07:55:51.559473377 +0000

stat exits successfully — /logs/init.d/sas.log exists inside the container.

Note

Root cause summary:

The container args ran stat /logs/init.d/sas.log without first creating the directory or file. Containers start with a minimal filesystem — /logs/init.d/ does not exist unless explicitly created. The pod appeared Running at all times because restartPolicy: Always restarted it immediately after each crash — but the RESTARTS counter revealed the crash loop. Adding mkdir -p /logs/init.d && touch /logs/init.d/sas.log before stat ensures the path exists, so stat succeeds and the container stays running.

Challenge progress

Tasks

Debug a Restarting Pod Using Previous Container Logs

Scenario

Task

Test Cases

Understanding the Solution

Step 1 : Check the pod status

Step 2 : Capture the previous container logs

Step 3 : Diagnose the issue from the logs

Step 4 : Edit the pod and fix the args

Step 5 : Verify the pod is Running with no restarts

Step 6 : Check the current pod logs

Step 7 : Verify the file exists inside the container