Fix a Deployment Stuck in ImagePullBackOff
Scenario
The catalog Deployment in the shop namespace was just rolled out, and its Pods do not start. From cplane-01:
kubectl get pods -n shop
NAME READY STATUS RESTARTS AGE
catalog-647b446c54-6l8bz 0/1 ImagePullBackOff 0 43s
catalog-647b446c54-jkmt5 0/1 ImagePullBackOff 0 43s
The catalog page at http://catalog.shop.local returns 503.
The team keeps the manifests and its notes on cplane-01:
/home/laborant/shop/
├── README.md
├── catalog-deployment.yaml
├── ingress.yaml # do not change
├── namespace.yaml # do not change
└── service.yaml # do not change
Task
Find out why the catalog Pods do not start, and get them running so that the catalog page loads at http://catalog.shop.local.
Keep the image of the catalog Deployment, registry.iximiuz.com/shop/catalog:v1, its imagePullPolicy, and its 2 replicas as they are.
Do not change the Service, the Ingress, or the configuration of the nodes.
Hint 1: Inspecting the failure
A Pod in ImagePullBackOff never started its container, so it has no logs yet. Its events show what the registry answered when the node tried to pull the image:
kubectl get pods -n <namespace>
kubectl describe pod <pod-name> -n <namespace>
Docs:
- Images: ImagePullBackOff (Kubernetes)
Hint 2: Reading the team's notes
The team keeps some notes alongside the manifests on cplane-01. Take a look at them and compare what you find with the error in the Pod events:
ls ~/shop/
cat ~/shop/README.md
Hint 3: Preparing the fix
kubectl create secret has a subcommand made for registry logins. Its help lists the flags for the registry address, username and password:
kubectl create secret --help
kubectl create secret <subcommand> --help
Check the Secret you created, including its type:
kubectl get secret <secret-name> -n <namespace>
Docs:
- Pull an Image from a Private Registry (Kubernetes)
Hint 4: Applying the fix
A Secret on its own does not affect image pulls. The Pods must be configured to use the registry credentials.
The credentials can be set directly in the Pod spec, or attached to the ServiceAccount that the Pods use. Inspect both to see how Kubernetes supplies image pull credentials:
kubectl explain pod.spec
kubectl get serviceaccount -n <namespace> -o yaml
Docs:
- Images: Using a private registry (Kubernetes)
Hint 5: Verifying the change
The old Pods do not pick up the change. Only new Pods use it.
A change to the Pod template in the Deployment starts a rollout on its own:
kubectl apply -f <manifest>
A change to the ServiceAccount does not touch the Deployment, so nothing replaces the Pods. Start a rollout yourself:
kubectl rollout restart deployment/<name> -n <namespace>
Then wait for the new Pods:
kubectl rollout status deployment/<name> -n <namespace>
kubectl get pods -n <namespace>
If a new Pod still fails to pull, its events show the registry's answer again. A wrong username or password also fails the pull, with 401 Unauthorized in the events.
Docs:
- Deployments: Updating a Deployment (Kubernetes)
- Debug Pods (Kubernetes)