Troubleshoot an Ingress That Returns 503 Service Unavailable
Scenario
The railway app in the railway namespace has two services, ticket and booking. An Ingress publishes them on one host:
| Endpoint | Backend |
|---|---|
http://railway.local/ticket | ticket |
http://railway.local/booking | booking |
Both endpoints fail. From cplane-01:
curl -s -i http://railway.local/ticket
HTTP/1.1 503 Service Unavailable
Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
Date: Mon, 12 Oct 2026 08:08:49 GMT
Content-Length: 20
no available server
curl -s -i http://railway.local/booking
HTTP/1.1 503 Service Unavailable
Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
Date: Mon, 12 Oct 2026 08:08:49 GMT
Content-Length: 20
no available server
The Deployments are applied and their Pods have started, but the app is not reachable.
The manifests the team applied are on cplane-01:
/home/laborant/railway/
├── booking-deployment.yaml
├── configmaps.yaml # immutable, do not change
├── ingress.yaml # do not change
├── services.yaml # do not change
└── ticket-deployment.yaml
Task
Make both endpoints answer with HTTP 200. Correct the Deployment manifests, ticket-deployment.yaml and booking-deployment.yaml in /home/laborant/railway/, and apply them.
Do not change the ConfigMaps, the Services or the Ingress. The ConfigMaps are immutable. Do not remove anything from the Deployments: fix what is wrong.
Hint 1 | Where the request stops
Start with the response itself: was the body written by the ingress controller or by the application?
Then follow the request from the Ingress to the Pods. In kubectl describe ingress, each backend lists in brackets the Pod addresses it can send traffic to. The EndpointSlices show whether those Pods are ready. A Pod that is Running does not always receive traffic:
kubectl describe ingress <ingress-name> -n <namespace>
kubectl get endpointslices -n <namespace>
kubectl get pods -n <namespace>
Hint 2 | Why the Pods are not ready
The events of a Pod show what the kubelet checks and what it gets back. Describe a single Pod by name:
kubectl describe pod <pod-name> -n <namespace>
The nginx logs show the same requests from the application's side. If a container keeps restarting, read the logs of its previous run:
kubectl logs <pod-name> -n <namespace>
kubectl logs <pod-name> -n <namespace> --previous
Hint 3 | What nginx really serves
The nginx.conf of each app is in an immutable ConfigMap. Its listen line and location blocks show the port and the paths nginx answers on. Compare them with what Kubernetes asks it for:
kubectl get configmap <name> -n <namespace> -o yaml
âš’ Test Cases
Understanding the challenge
A request to http://railway.local/ticket or http://railway.local/booking goes through Traefik, the Ingress, a Service, and finally a Pod:
client --> Traefik (port 80) --> Ingress --> Service ticket --> ticket Pods (nginx on 8080)
--> Service booking --> booking Pods (nginx on 8080)
Both endpoints answer 503 Service Unavailable, yet the Pods exist and their containers start. The ticket Pods stay Running, while the booking Pods keep restarting and end up in CrashLoopBackOff within about a minute. The Ingress, the Services and the nginx config are correct. The problem is in how the Deployments check the health of their Pods: the probes.
Running is not the same as Ready
A Pod can be Running and still receive no traffic. Kubernetes uses probes to decide:
| Probe | Question it asks | If it fails |
|---|---|---|
| Readiness probe | Can this container serve requests right now? | The Pod is marked not ready. It keeps running, but the Service stops sending it traffic. |
| Liveness probe | Is this container still alive? | The kubelet kills and restarts the container. Repeated restarts end in CrashLoopBackOff. |
A Service sends traffic only to ready Pods. A not-ready Pod does not disappear from the Service's EndpointSlice: it stays listed, with the condition ready: false. Being listed and being eligible for traffic are two different things. Only endpoints with ready: true are eligible. When no Pod behind a Service is ready, Traefik has nowhere to send the request and answers:
HTTP/1.1 503 Service Unavailable
no available server
So a 503 with Running Pods points to a readiness problem: the Pods exist, but none of them passed the readiness probe.
What a 503 means, and who sent it
A 503 Service Unavailable can come from two different places, and they mean different things:
| Who answers 503 | What it means | How to recognise it |
|---|---|---|
| The ingress controller (Traefik) | Traefik found the route, but has no usable backend for it, most often because the Service has no ready Pod. It did not send the request to any Pod at all. | Plain text body no available server. The Ingress backend shows (). EndpointSlices show Ready: false or no addresses. |
| The application | The request did reach a ready Pod, and the application itself refused it, for example because it is overloaded or a database it needs is down. | The body comes from the app (an nginx HTML page, a JSON error, ...). The Pods are ready, and the app logs show the request. |
A 503 with no available server does not mean "the Pod was reached but did not answer". When Traefik does send the request to a Pod and that fails, you usually get a different code: typically 502 Bad Gateway when the connection is refused, or 504 Gateway Timeout when the Pod does not reply in time. The exact code depends on the proxy and its settings, so always read the response body and, if needed, the ingress controller logs. In the 404 challenge on this cluster, pointing a Service at a port where nothing listens gave a 502.
Traefik can also answer 503 for other reasons, for example from a middleware such as a circuit breaker. In this challenge three pieces of evidence point the same way: the body is no available server, the Ingress backends are empty, and the readiness probes fail. Together they show that Traefik has no ready Pod to send traffic to.
Common causes of an ingress 503
Anything that leaves a Service without a ready Pod produces the same Traefik 503. The probes are only one of the causes:
| Cause | What you see on the Pods |
|---|---|
| Readiness probe misconfigured (wrong path or port) | Running, 0/1, no restarts, events Readiness probe failed |
| Liveness probe misconfigured | Restarts, CrashLoopBackOff, events failed liveness probe, will be restarted |
| Application crashes on start (bad config, missing file, bad command) | CrashLoopBackOff, Last State: Terminated, Reason: Error, errors in kubectl logs --previous |
| Memory leak or memory limit too low | Last State: Terminated, Reason: OOMKilled, exit code 137, restarts |
| Application starts slowly (readiness checked before it is up) | 0/1 for a while after every start, then ready |
No Pods selected (Service selector wrong, replicas 0) | EndpointSlice with no addresses at all |
A common interview question: "Your application behind an Ingress returns 503. What could cause it, and how do you find out?"
There is no single answer, because many different problems end in the same 503. A good answer walks the request path instead of guessing:
- Who sent the 503? The ingress controller (
no available server) or the application itself (its own error page or JSON)? - Does the Service have ready endpoints?
kubectl get endpointslicesandkubectl describe ingressshow whether any Pod can receive traffic. - Why are the Pods not ready?
kubectl get podsandkubectl describe podshow probe failures, restarts,CrashLoopBackOfforOOMKilled. - If the Pods are ready, the 503 comes from the application: check its logs and the services it depends on.
Naming several causes from the table above, and explaining how you would tell them apart, shows you understand how traffic reaches a Pod.
In this challenge, both Deployments have probe misconfigurations. The application itself is fine: nginx starts, listens on 8080 and answers its health paths. Only the probes ask it the wrong questions.
What a probe must match
An HTTP probe asks the container for a path on a port. It passes only when the answer is a 2xx or 3xx status code. Both values must match what the application really serves:
| Probe field | Must match | Defined in nginx by |
|---|---|---|
httpGet.port | The port the application listens on | the listen line in nginx.conf |
httpGet.path | A path that answers 200 | the location blocks in nginx.conf |
Step 1: Reproduce the error
The output in these steps comes from a real run. Your Pod names, IP addresses, ages and dates will be different.
curl -s -i http://railway.local/ticket
HTTP/1.1 503 Service Unavailable
Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
Date: Fri, 16 Oct 2026 19:29:37 GMT
Content-Length: 20
no available server
curl -s -i http://railway.local/booking
HTTP/1.1 503 Service Unavailable
Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
Date: Fri, 16 Oct 2026 19:29:37 GMT
Content-Length: 20
no available server
The plain text body no available server is written by Traefik, not by nginx. There is no Server: nginx header, so the request never reached a Pod.
Step 2: Check the Deployments and Pods
kubectl get deployment,pods -n railway
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/booking 0/2 2 0 41s
deployment.apps/ticket 0/2 2 0 41s
NAME READY STATUS RESTARTS AGE
pod/booking-b4d8f6cd-2sb89 0/1 Running 2 (5s ago) 41s
pod/booking-b4d8f6cd-wqbdj 0/1 Running 2 (4s ago) 41s
pod/ticket-6b44464f48-x9s9x 0/1 Running 0 41s
pod/ticket-6b44464f48-z7h8n 0/1 Running 0 41s
All four Pods are Running, but READY is 0/1 for each of them. The booking Pods have already restarted twice in less than a minute, and they soon end up in CrashLoopBackOff, as Step 6 shows. The ticket Pods do not restart.
Step 3: See why the Ingress has no backend
kubectl describe ingress railway -n railway
Name: railway
Labels: <none>
Namespace: railway
Address: 172.16.0.2,172.16.0.3,172.16.0.4
Ingress Class: traefik
Default backend: <default>
Rules:
Host Path Backends
---- ---- --------
railway.local
/ticket ticket:80 ()
/booking booking:80 ()
Annotations: <none>
Events: <none>
The empty () means no ready address behind either Service. The short EndpointSlice view lists the Pod addresses, but does not show whether they are ready:
kubectl get endpointslices -n railway
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
booking-zh24r IPv4 8080 10.42.3.4,10.42.0.9 41s
ticket-sqksl IPv4 8080 10.42.3.3,10.42.1.4 41s
The detailed view shows that they are not ready:
kubectl describe endpointslice -n railway -l kubernetes.io/service-name=ticket
Name: ticket-sqksl
Namespace: railway
Labels: endpointslice.kubernetes.io/managed-by=endpointslice-controller.k8s.io
kubernetes.io/service-name=ticket
Annotations: <none>
AddressType: IPv4
Ports:
Name Port Protocol
---- ---- --------
<unset> 8080 TCP
Endpoints:
- Addresses: 10.42.3.3
Conditions:
Ready: false
Hostname: <unset>
TargetRef: Pod/ticket-6b44464f48-x9s9x
NodeName: node-01
Zone: <unset>
- Addresses: 10.42.1.4
Conditions:
Ready: false
Hostname: <unset>
TargetRef: Pod/ticket-6b44464f48-z7h8n
NodeName: node-02
Zone: <unset>
Events: <none>
Both Pods are still listed as endpoints, but with Ready: false. They are known to the Service, but not eligible for traffic, so Traefik has no backend to use. The Services found the right Pods on port 8080, so the selectors and ports are fine. The Pods are simply not ready.
Step 4: Investigate the ticket Pods
kubectl get pods -n railway -l app=ticket
NAME READY STATUS RESTARTS AGE
ticket-6b44464f48-x9s9x 0/1 Running 0 53s
ticket-6b44464f48-z7h8n 0/1 Running 0 53s
Describe one of the Pods by name. Use your own Pod name from the output above:
kubectl describe pod ticket-6b44464f48-x9s9x -n railway
Name: ticket-6b44464f48-x9s9x
Namespace: railway
Priority: 0
Service Account: default
Node: node-01/172.16.0.3
Start Time: Fri, 16 Oct 2026 19:29:25 +0000
Labels: app=ticket
pod-template-hash=6b44464f48
Annotations: <none>
Status: Running
IP: 10.42.3.3
IPs:
IP: 10.42.3.3
Controlled By: ReplicaSet/ticket-6b44464f48
Containers:
nginx:
Container ID: containerd://d49e6c1ac9919b6988f21742a590702ee25f91ff709b730173c075fce1365bdd
Image: nginx:1.30-alpine
Image ID: docker.io/library/nginx@sha256:970364228b1e4f932abc22d321932816782cca8a20b85a77552a7cfe9acfb3b5
Port: 8080/TCP
Host Port: 0/TCP
State: Running
Started: Fri, 16 Oct 2026 19:29:31 +0000
Ready: False
Restart Count: 0
Readiness: http-get http://:8080/health delay=0s timeout=1s period=5s successThreshold=1 failureThreshold=3
Environment: <none>
Mounts:
/etc/nginx/conf.d from conf (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-d2jvs (ro)
Conditions:
Type Status
PodReadyToStartContainers True
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
conf:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: ticket-nginx
Optional: false
kube-api-access-d2jvs:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
Optional: false
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 52s default-scheduler Successfully assigned railway/ticket-6b44464f48-x9s9x to node-01
Normal Pulling 52s kubelet spec.containers{nginx}: Pulling image "nginx:1.30-alpine"
Normal Pulled 47s kubelet spec.containers{nginx}: Successfully pulled image "nginx:1.30-alpine" in 5.276s (5.276s including waiting). Image size: 28545847 bytes.
Normal Created 47s kubelet spec.containers{nginx}: Container created
Normal Started 47s kubelet spec.containers{nginx}: Container started
Warning Unhealthy 47s kubelet spec.containers{nginx}: Readiness probe failed: Get "http://10.42.3.3:8080/health": dial tcp 10.42.3.3:8080: connect: connection refused
Warning Unhealthy 5s (x10 over 46s) kubelet spec.containers{nginx}: Readiness probe failed: HTTP probe failed with statuscode: 404
Read the Readiness: line and the last two events:
- A
connection refusedright after the container starts can happen while nginx is still opening port8080. It may or may not appear in your run, and it is not the problem. - Every probe after that reaches nginx and gets
404. The probe asks for/health, but nginx serves/healthz. - The repeated
404is the real evidence. The port8080is correct. There is no liveness probe, so the kubelet has no reason to restart the container. It just stays at0/1.
Describe a single Pod by name when you need its probe events. On this cluster (kubectl v1.37), kubectl describe pod -l app=ticket printed the Pods without an Events section, and the events are exactly what explains a failing probe.
The events show what the kubelet saw. The nginx logs show the same probe from the application's side:
kubectl logs ticket-6b44464f48-x9s9x -n railway --tail=8
2026/10/16 19:29:58 [error] 22#22: *7 open() "/etc/nginx/html/health" failed (2: No such file or directory), client: 10.42.3.1, server: , request: "GET /health HTTP/1.1", host: "10.42.3.3:8080"
10.42.3.1 - - [16/Oct/2026:19:29:58 +0000] "GET /health HTTP/1.1" 404 153 "-" "kube-probe/1.37" "-"
2026/10/16 19:30:03 [error] 22#22: *8 open() "/etc/nginx/html/health" failed (2: No such file or directory), client: 10.42.3.1, server: , request: "GET /health HTTP/1.1", host: "10.42.3.3:8080"
10.42.3.1 - - [16/Oct/2026:19:30:03 +0000] "GET /health HTTP/1.1" 404 153 "-" "kube-probe/1.37" "-"
10.42.3.1 - - [16/Oct/2026:19:30:08 +0000] "GET /health HTTP/1.1" 404 153 "-" "kube-probe/1.37" "-"
2026/10/16 19:30:08 [error] 22#22: *9 open() "/etc/nginx/html/health" failed (2: No such file or directory), client: 10.42.3.1, server: , request: "GET /health HTTP/1.1", host: "10.42.3.3:8080"
10.42.3.1 - - [16/Oct/2026:19:30:13 +0000] "GET /health HTTP/1.1" 404 153 "-" "kube-probe/1.37" "-"
2026/10/16 19:30:13 [error] 22#22: *10 open() "/etc/nginx/html/health" failed (2: No such file or directory), client: 10.42.3.1, server: , request: "GET /health HTTP/1.1", host: "10.42.3.3:8080"
| Log line | What it tells you |
|---|---|
"GET /health HTTP/1.1" 404 ... "kube-probe/1.37" | The request came from the kubelet's probe (kube-probe user agent) every 5 seconds, and nginx answered 404. |
open() "/etc/nginx/html/health" failed | No location in the config matches /health, so nginx fell back to looking for a file of that name, and there is none. |
So the probe does reach nginx on the right port, but nginx has nothing at /health. To see what nginx does serve, look at its configuration.
Read the ticket nginx.conf
The ticket app's nginx config, its nginx.conf server block, is mounted from the ticket-nginx ConfigMap:
kubectl get configmap ticket-nginx -n railway -o yaml
apiVersion: v1
data:
default.conf: |
server {
listen 8080;
location = /healthz {
return 200 "ok\n";
}
location /ticket {
default_type application/json;
return 200 '{"service": "ticket", "status": "available"}\n';
}
}
immutable: true
kind: ConfigMap
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","data":{"default.conf":"server {\n listen 8080;\n\n location = /healthz {\n return 200 \"ok\\n\";\n }\n\n location /ticket {\n default_type application/json;\n return 200 '{\"service\": \"ticket\", \"status\": \"available\"}\\n';\n }\n}\n"},"immutable":true,"kind":"ConfigMap","metadata":{"annotations":{},"name":"ticket-nginx","namespace":"railway"}}
creationTimestamp: "2026-10-16T19:29:25Z"
name: ticket-nginx
namespace: railway
resourceVersion: "841"
uid: c3807321-97d1-4090-8315-45ea9881c0cc
nginx listens on 8080, and its location blocks serve /healthz and /ticket. There is no location for /health, which is exactly why the probe gets 404 and the log shows nginx looking for a file instead.
The ConfigMap also carries immutable: true. Kubernetes rejects any change to the data of an immutable ConfigMap:
kubectl patch configmap ticket-nginx -n railway --type merge -p '{"data":{"default.conf":"server {}"}}'
The ConfigMap "ticket-nginx" is invalid: data: Forbidden: field is immutable when `immutable` is set
The only way to change an immutable ConfigMap is to delete it and create it again. That is a deliberate, visible step, and the task forbids it here.
Fix the probes, not nginx.conf. It can look tempting to make nginx answer what the probes ask for, for example to add a /health location for ticket, or a listen 80; line for booking. That would be the wrong fix:
- The nginx config describes what the application really serves. The probes are Kubernetes asking the application questions, so the probes must follow the application, not the other way round.
- Changing the application to match a broken probe hides the real mistake in the Deployment, and the next Deployment written the same way breaks again.
- Opening an extra port or path changes the application for every client, not only for the kubelet.
That is why both ConfigMaps are immutable in this challenge: the nginx config is correct, and the fix belongs in the Deployments.
Step 5: Fix the ticket readiness probe
cat /home/laborant/railway/ticket-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: ticket
namespace: railway
spec:
replicas: 2
selector:
matchLabels:
app: ticket
template:
metadata:
labels:
app: ticket
spec:
containers:
- name: nginx
image: nginx:1.30-alpine
ports:
- containerPort: 8080
readinessProbe:
httpGet:
path: /health
port: 8080
periodSeconds: 5
volumeMounts:
- name: conf
mountPath: /etc/nginx/conf.d
volumes:
- name: conf
configMap:
name: ticket-nginx
Change the path to /healthz:
vi /home/laborant/railway/ticket-deployment.yaml
readinessProbe:
httpGet:
path: /healthz # the health path nginx serves
port: 8080
periodSeconds: 5
kubectl apply -f /home/laborant/railway/ticket-deployment.yaml
deployment.apps/ticket configured
kubectl rollout status deployment/ticket -n railway
Waiting for deployment "ticket" rollout to finish: 0 out of 2 new replicas have been updated...
Waiting for deployment "ticket" rollout to finish: 1 out of 2 new replicas have been updated...
Waiting for deployment "ticket" rollout to finish: 1 out of 2 new replicas have been updated...
Waiting for deployment "ticket" rollout to finish: 1 out of 2 new replicas have been updated...
Waiting for deployment "ticket" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "ticket" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "ticket" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "ticket" rollout to finish: 1 old replicas are pending termination...
deployment "ticket" successfully rolled out
curl -s -i http://railway.local/ticket
HTTP/1.1 200 OK
Content-Length: 45
Content-Type: application/json
Date: Fri, 16 Oct 2026 19:30:26 GMT
Server: nginx/1.30.6
{"service": "ticket", "status": "available"}
This time the answer has a Server: nginx/1.30.6 header: the request reached a ticket Pod. The logs of the new Pods now show the probe succeeding:
kubectl logs deployment/ticket -n railway --tail=2
Found 2 pods, using pod/ticket-6b7674f9cd-8fdgk
10.42.0.1 - - [16/Oct/2026:19:31:04 +0000] "GET /healthz HTTP/1.1" 200 3 "-" "kube-probe/1.37" "-"
10.42.0.1 - - [16/Oct/2026:19:31:09 +0000] "GET /healthz HTTP/1.1" 200 3 "-" "kube-probe/1.37" "-"
The booking endpoint still answers 503.
Step 6: Investigate the booking Pods
kubectl get pods -n railway -l app=booking
NAME READY STATUS RESTARTS AGE
booking-b4d8f6cd-2sb89 0/1 CrashLoopBackOff 3 (6s ago) 72s
booking-b4d8f6cd-wqbdj 0/1 CrashLoopBackOff 3 (5s ago) 72s
Both booking Pods have already restarted 3 times. CrashLoopBackOff means the kubelet is waiting before the next restart. Between restarts a Pod briefly shows Running again, so the status switches back and forth.
Describe one of the Pods by name:
kubectl describe pod booking-b4d8f6cd-2sb89 -n railway
Name: booking-b4d8f6cd-2sb89
Namespace: railway
Priority: 0
Service Account: default
Node: cplane-01/172.16.0.2
Start Time: Fri, 16 Oct 2026 19:29:25 +0000
Labels: app=booking
pod-template-hash=b4d8f6cd
Annotations: <none>
Status: Running
IP: 10.42.0.9
IPs:
IP: 10.42.0.9
Controlled By: ReplicaSet/booking-b4d8f6cd
Containers:
nginx:
Container ID: containerd://5ad88f9aab427ca775b0b9a716331a71aa7c83dec8da07e590d171e2c16d1625
Image: nginx:1.30-alpine
Image ID: docker.io/library/nginx@sha256:970364228b1e4f932abc22d321932816782cca8a20b85a77552a7cfe9acfb3b5
Port: 8080/TCP
Host Port: 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Completed
Exit Code: 0
Started: Fri, 16 Oct 2026 19:30:16 +0000
Finished: Fri, 16 Oct 2026 19:30:31 +0000
Ready: False
Restart Count: 3
Liveness: http-get http://:80/ready delay=5s timeout=1s period=5s successThreshold=1 failureThreshold=3
Readiness: http-get http://:80/ready delay=0s timeout=1s period=5s successThreshold=1 failureThreshold=3
Environment: <none>
Mounts:
/etc/nginx/conf.d from conf (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-cmdwk (ro)
Conditions:
Type Status
PodReadyToStartContainers True
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
conf:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: booking-nginx
Optional: false
kube-api-access-cmdwk:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
Optional: false
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 71s default-scheduler Successfully assigned railway/booking-b4d8f6cd-2sb89 to cplane-01
Normal Pulling 71s kubelet spec.containers{nginx}: Pulling image "nginx:1.30-alpine"
Normal Pulled 66s kubelet spec.containers{nginx}: Successfully pulled image "nginx:1.30-alpine" in 5.302s (5.302s including waiting). Image size: 28545847 bytes.
Normal Created 21s (x4 over 66s) kubelet spec.containers{nginx}: Container created
Normal Started 21s (x4 over 66s) kubelet spec.containers{nginx}: Container started
Warning Unhealthy 21s (x16 over 66s) kubelet spec.containers{nginx}: Readiness probe failed: Get "http://10.42.0.9:80/ready": dial tcp 10.42.0.9:80: connect: connection refused
Warning Unhealthy 21s (x9 over 61s) kubelet spec.containers{nginx}: Liveness probe failed: Get "http://10.42.0.9:80/ready": dial tcp 10.42.0.9:80: connect: connection refused
Normal Pulled 21s (x3 over 51s) kubelet spec.containers{nginx}: Container image "nginx:1.30-alpine" already present on machine and can be accessed by the pod
Normal Killing 6s (x4 over 51s) kubelet spec.containers{nginx}: Container nginx failed liveness probe, will be restarted
The path /ready is correct, but both probes knock on port 80, where nothing listens: connection refused. Two things happen at once:
- The readiness probe fails, so the Pod is never ready and gets no traffic.
- The liveness probe fails, so the kubelet keeps killing and restarting nginx (
Killing ... will be restarted), which ends inCrashLoopBackOff.
Look at Last State: Reason: Completed, Exit Code: 0. nginx did not crash. The kubelet stopped it cleanly after the liveness probe failed. A real application crash would show Reason: Error or OOMKilled with a non-zero exit code.
The logs confirm it. Because the container keeps restarting, read the logs of the previous run with --previous:
kubectl logs booking-b4d8f6cd-2sb89 -n railway --previous
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: can not modify /etc/nginx/conf.d/default.conf (read-only file system?)
/docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2026/10/16 19:30:16 [notice] 1#1: using the "epoll" event method
2026/10/16 19:30:16 [notice] 1#1: nginx/1.30.6
2026/10/16 19:30:16 [notice] 1#1: built by gcc 15.2.0 (Alpine 15.2.0)
2026/10/16 19:30:16 [notice] 1#1: OS: Linux 6.1.167
2026/10/16 19:30:16 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2026/10/16 19:30:16 [notice] 1#1: start worker processes
2026/10/16 19:30:16 [notice] 1#1: start worker process 21
2026/10/16 19:30:16 [notice] 1#1: start worker process 22
2026/10/16 19:30:16 [notice] 1#1: start worker process 23
2026/10/16 19:30:16 [notice] 1#1: start worker process 24
2026/10/16 19:30:31 [notice] 1#1: signal 3 (SIGQUIT) received, shutting down
2026/10/16 19:30:31 [notice] 21#21: gracefully shutting down
2026/10/16 19:30:31 [notice] 22#22: gracefully shutting down
2026/10/16 19:30:31 [notice] 24#24: gracefully shutting down
2026/10/16 19:30:31 [notice] 22#22: exiting
2026/10/16 19:30:31 [notice] 21#21: exiting
2026/10/16 19:30:31 [notice] 24#24: exiting
2026/10/16 19:30:31 [notice] 24#24: exit
2026/10/16 19:30:31 [notice] 22#22: exit
2026/10/16 19:30:31 [notice] 21#21: exit
2026/10/16 19:30:31 [notice] 23#23: gracefully shutting down
2026/10/16 19:30:31 [notice] 23#23: exiting
2026/10/16 19:30:31 [notice] 23#23: exit
2026/10/16 19:30:31 [notice] 1#1: signal 17 (SIGCHLD) received from 21
2026/10/16 19:30:31 [notice] 1#1: worker process 21 exited with code 0
2026/10/16 19:30:31 [notice] 1#1: worker process 22 exited with code 0
2026/10/16 19:30:31 [notice] 1#1: worker process 23 exited with code 0
2026/10/16 19:30:31 [notice] 1#1: signal 29 (SIGIO) received
2026/10/16 19:30:31 [notice] 1#1: signal 17 (SIGCHLD) received from 22
2026/10/16 19:30:31 [notice] 1#1: signal 17 (SIGCHLD) received from 24
2026/10/16 19:30:31 [notice] 1#1: worker process 24 exited with code 0
2026/10/16 19:30:31 [notice] 1#1: exit
Compare this with the ticket logs in Step 4:
| What the log shows | What it means |
|---|---|
| nginx starts its workers and reports no error | The config is valid and nginx is listening on 8080. |
No GET /ready line at all | No probe ever reached nginx. The probes go to port 80, where nothing listens, so the connection is refused before nginx sees a request. |
signal 3 (SIGQUIT) received, shutting down about 15 seconds later | The kubelet asked nginx to stop after the liveness probe failed 3 times. nginx shut down cleanly with exit code 0. |
Read the booking nginx.conf
The booking app's nginx config is mounted from the booking-nginx ConfigMap, which is immutable too:
kubectl get configmap booking-nginx -n railway -o yaml
apiVersion: v1
data:
default.conf: |
server {
listen 8080;
location = /ready {
return 200 "ok\n";
}
location /booking {
default_type application/json;
return 200 '{"service": "booking", "status": "open"}\n';
}
}
immutable: true
kind: ConfigMap
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","data":{"default.conf":"server {\n listen 8080;\n\n location = /ready {\n return 200 \"ok\\n\";\n }\n\n location /booking {\n default_type application/json;\n return 200 '{\"service\": \"booking\", \"status\": \"open\"}\\n';\n }\n}\n"},"immutable":true,"kind":"ConfigMap","metadata":{"annotations":{},"name":"booking-nginx","namespace":"railway"}}
creationTimestamp: "2026-10-16T19:29:25Z"
name: booking-nginx
namespace: railway
resourceVersion: "842"
uid: 86c48668-2a29-4301-8e4f-4ddda5bfe214
nginx listens on 8080 and serves /ready. The path in both probes is right. The port is not: the probes check port 80, where nginx does not listen. As with ticket, the fix belongs in the Deployment, not in nginx.conf.
Step 7: Fix the booking probes
containerPort: 8080 only documents the port. It does not make nginx listen on 8080 (the listen line in the ConfigMap does that), and a probe does not take its port from it automatically. Each probe checks the port set in its own httpGet.port: a number, or the name of a named container port. That is why the booking probes fail on port 80 even though the Deployment declares containerPort: 8080.
cat /home/laborant/railway/booking-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: booking
namespace: railway
spec:
replicas: 2
selector:
matchLabels:
app: booking
template:
metadata:
labels:
app: booking
spec:
containers:
- name: nginx
image: nginx:1.30-alpine
ports:
- containerPort: 8080
livenessProbe:
httpGet:
path: /ready
port: 80
initialDelaySeconds: 5
periodSeconds: 5
failureThreshold: 3
readinessProbe:
httpGet:
path: /ready
port: 80
periodSeconds: 5
volumeMounts:
- name: conf
mountPath: /etc/nginx/conf.d
volumes:
- name: conf
configMap:
name: booking-nginx
Change both probe ports to 8080:
vi /home/laborant/railway/booking-deployment.yaml
livenessProbe:
httpGet:
path: /ready
port: 8080 # the port nginx listens on
initialDelaySeconds: 5
periodSeconds: 5
failureThreshold: 3
readinessProbe:
httpGet:
path: /ready
port: 8080 # the port nginx listens on
periodSeconds: 5
kubectl apply -f /home/laborant/railway/booking-deployment.yaml
deployment.apps/booking configured
kubectl rollout status deployment/booking -n railway
Waiting for deployment "booking" rollout to finish: 1 out of 2 new replicas have been updated...
Waiting for deployment "booking" rollout to finish: 1 out of 2 new replicas have been updated...
Waiting for deployment "booking" rollout to finish: 1 out of 2 new replicas have been updated...
Waiting for deployment "booking" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "booking" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "booking" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "booking" rollout to finish: 1 old replicas are pending termination...
deployment "booking" successfully rolled out
The new Pods now log the probes, and nginx answers them with 200:
kubectl logs deployment/booking -n railway --tail=2
Found 2 pods, using pod/booking-755d6d5f94-g8jvv
10.42.3.1 - - [16/Oct/2026:19:31:11 +0000] "GET /ready HTTP/1.1" 200 3 "-" "kube-probe/1.37" "-"
10.42.3.1 - - [16/Oct/2026:19:31:12 +0000] "GET /ready HTTP/1.1" 200 3 "-" "kube-probe/1.37" "-"
Two /ready requests arrive close together because both the liveness and the readiness probe now reach nginx.
Step 8: Confirm both endpoints work
kubectl get deployment,pods -n railway
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/booking 2/2 2 2 96s
deployment.apps/ticket 2/2 2 2 96s
NAME READY STATUS RESTARTS AGE
pod/booking-755d6d5f94-g8jvv 1/1 Running 0 15s
pod/booking-755d6d5f94-pdgq5 1/1 Running 0 14s
pod/ticket-6b7674f9cd-8fdgk 1/1 Running 0 37s
pod/ticket-6b7674f9cd-s5fd8 1/1 Running 0 37s
kubectl describe ingress railway -n railway
Name: railway
Labels: <none>
Namespace: railway
Address: 172.16.0.2,172.16.0.3,172.16.0.4
Ingress Class: traefik
Default backend: <default>
Rules:
Host Path Backends
---- ---- --------
railway.local
/ticket ticket:80 (10.42.0.10:8080,10.42.1.5:8080)
/booking booking:80 (10.42.3.5:8080,10.42.1.6:8080)
Annotations: <none>
Events: <none>
Both backends now list ready Pod addresses.
curl -s -i http://railway.local/ticket
HTTP/1.1 200 OK
Content-Length: 45
Content-Type: application/json
Date: Fri, 16 Oct 2026 19:31:01 GMT
Server: nginx/1.30.6
{"service": "ticket", "status": "available"}
curl -s -i http://railway.local/booking
HTTP/1.1 200 OK
Content-Length: 41
Content-Type: application/json
Date: Fri, 16 Oct 2026 19:31:01 GMT
Server: nginx/1.30.6
{"service": "booking", "status": "open"}
The /ticket and /booking tabs now show the JSON answers.
Summary
| App | What you saw | Root cause | Fix |
|---|---|---|---|
| ticket | 0/1 Running, no restarts, Readiness probe failed ... statuscode: 404 | Readiness probe asks for /health, nginx serves /healthz | Readiness path /healthz |
| booking | Restarts, CrashLoopBackOff, connection refused on port 80 | Liveness and readiness probes check port 80, nginx listens on 8080 | Both probe ports 8080 |
In both cases the Pods were not ready, so Traefik had no eligible backend and answered 503 with no available server. Once the probes matched what nginx really serves, the Pods became ready and both routes answered 200.
Do not "fix" a failing probe by deleting it. Without a readiness probe, the Pod gets traffic even while nginx is still starting or broken. Without a liveness probe, a hung nginx is not restarted, because nothing checks that it still answers. The check requires the probes to stay, so point them at the right path and port instead.
Other probe settings can also work. For example, the ticket readiness probe could ask for /ticket, which also answers 200. A dedicated health path like /healthz is the better choice: it is cheap to answer and does not depend on the business logic of the endpoint.
Want to practise the other common ingress error? In Troubleshoot an Ingress That Returns 404 Not Found, Traefik has no route at all, because the Ingress points at a Service port that does not exist, and the Service selects no Pods.