Troubleshoot an Ingress That Returns 404 Not Found
Scenario
The bose-einstein Deployment runs nginx in the physics namespace. It is exposed through the bose-einstein Ingress at http://bose.physics.local, but the address returns 404 page not found.
nginx in the Pods is healthy and listens on port 9090. The files the team applied are on cplane-01. Only ingress.yaml and service.yaml are misconfigured. Do not change the other files:
/home/laborant/bose-einstein/
├── default.conf # do not change
├── deployment.yaml # do not change
├── index.html # do not change
├── ingress.yaml # misconfigured
└── service.yaml # misconfigured
Task
Make the necessary changes to the misconfigured files in /home/laborant/bose-einstein/ and apply them, so that the application is reachable at http://bose.physics.local.
Do not change or recreate the bose-einstein Deployment. Keep the Ingress host bose.physics.local, and do not change Traefik.
Hint 1 | Following a request from the Ingress to the Pod
A request passes through the Ingress, then the Service, then a Pod. Check each link in turn:
kubectl describe ingress <name> -n <namespace>
kubectl describe service <name> -n <namespace>
kubectl get endpointslices -n <namespace>
kubectl get pods -n <namespace> --show-labels
Hint 2 | Which port goes where
Three different fields hold a port, and each one points to the next:
| Field | Points to |
|---|---|
Ingress backend.service.port | A port of the Service |
Service port | The port the Service exposes |
Service targetPort | The port the container listens on |
âš’ Test Cases
Understanding the challenge
When you open http://bose.physics.local, the request travels through four objects before nginx answers:
client --> Traefik (port 80) --> Ingress --> Service --> Pod (nginx on 9090)
Each object has to point correctly at the next one. If one link is wrong, the request stops there. The Pods are running and nginx works, so the problem is in the links.
Only two files are misconfigured, service.yaml and ingress.yaml. The Deployment must not be changed or recreated, so the values it defines are fixed, and everything else has to match them.
Two different selectors
Both the Deployment and the Service have a selector, but they do different jobs.
Deployment selector: "which Pods do I own?"
The Deployment uses its selector to find and manage its own Pods: count them, replace them, roll them out. In deployment.yaml it has only one label:
spec:
selector:
matchLabels:
app: bose-einstein # the Deployment looks for this one label
template:
metadata:
labels:
app: bose-einstein # every Pod gets these two labels
tier: frontend
Every Pod gets two labels from the template, app and tier. The Deployment only needs app to recognise them as its own. The extra tier label is there for others to use.
Service selector: "which Pods do I send traffic to?"
The Service uses its selector to decide which Pods receive traffic. A Pod is picked only if it has every label in the Service selector, with exactly the same value. The picked Pods become the Service's endpoints.
In this challenge the Service selects on two labels, app and tier, so a Pod must carry both.
| Deployment selector | Service selector | |
|---|---|---|
| Purpose | Find the Pods it creates and manages | Find the Pods to send traffic to |
| Labels here | app: bose-einstein | app: bose-einstein, tier: frontend |
| If it matches nothing | The Deployment cannot manage its Pods | The Service has no endpoints, nobody receives traffic |
Because the two selectors are independent, the Deployment can show 2/2 Pods ready while the Service finds none of them.
What is fixed and what is free
The Deployment cannot change, so two values are fixed. The Service must match them exactly:
| Fixed value | Comes from | Must be set in |
|---|---|---|
Pod labels app: bose-einstein, tier: frontend | Deployment Pod template | Service selector |
nginx port 9090 | default.conf (listen 9090;), shown as containerPort in the Deployment | Service targetPort |
The rest is free. Three port fields are involved, and each one only has to agree with the next:
| Field | Meaning | Rule |
|---|---|---|
Ingress backend.service.port | Which Service port the Ingress sends to | Must equal a Service port |
Service port | The port the Service listens on | Any value |
Service targetPort | The Pod port the Service forwards to | Must be 9090 |
So the Service port and the Ingress port can be any number, as long as they are the same. This gives several correct solutions, shown at the end.
Where does 9090 really come from? Not from containerPort. That field only documents the port in the Pod spec. nginx listens on 9090 because its config file default.conf says listen 9090;, and the readiness probe checks that port. The targetPort must match the port the application really listens on. Changing containerPort alone would not move nginx to another port.
What the error code tells you
The HTTP status code is a useful clue about which link is broken. It is not proof on its own: other controllers, and other mistakes, can produce the same codes. The table shows the usual causes:
| Response | Usual causes | Look at |
|---|---|---|
404 Not Found | No Ingress route matches: wrong host or path, or the backend points at a Service or port that does not exist. The application itself can also answer 404. | Ingress host, path, backend Service and port |
503 Service Unavailable | The route exists, but the controller has no ready backend: the Service selects no Pods, or the Pods are not ready. | Service selector, ready EndpointSlices |
502 Bad Gateway | The controller reaches a Pod address, but the connection fails, for example because nothing listens on the target port. | Service targetPort, endpoint ports, the application |
200 OK | The request reached the application and it answered. | Check the response is the expected page |
In this cluster, Traefik 3.7 answered with these exact responses at each stage of the challenge:
| Stage | Response |
|---|---|
| Start, all three fields wrong | 404 page not found |
| Only the Ingress port fixed | 503 Service Unavailable, no available server |
| Ingress port and selector fixed | 502 Bad Gateway |
| Everything fixed | 200 OK |
Who wrote the 404 matters. 404 page not found as plain text comes from Traefik, which means the request never reached nginx. A 404 page from nginx would mean the routing works and the requested file is missing.
Step 1: Reproduce the error
curl -s http://bose.physics.local/
404 page not found
kubectl get deployment,service,ingress -n physics
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/bose-einstein 2/2 2 2 44s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/bose-einstein ClusterIP 10.43.67.63 <none> 9090/TCP 44s
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress.networking.k8s.io/bose-einstein traefik bose.physics.local 172.16.0.2,172.16.0.3,172.16.0.4 80 44s
Everything exists and the Deployment is 2/2. The PORTS column of the Ingress (80) is where Traefik listens, not the backend port.
Step 2: Check the Ingress
kubectl describe ingress bose-einstein -n physics
Name: bose-einstein
Labels: <none>
Namespace: physics
Address: 172.16.0.2,172.16.0.3,172.16.0.4
Ingress Class: traefik
Default backend: <default>
Rules:
Host Path Backends
---- ---- --------
bose.physics.local
/ bose-einstein:80 ()
Annotations: <none>
Events: <none>
bose-einstein:80 means "send to port 80 of the Service". The empty () means nothing is behind it. The Service listens on 9090, not 80, so this is the first broken link.
Step 3: Check the Service
kubectl describe service bose-einstein -n physics
Name: bose-einstein
Namespace: physics
Labels: <none>
Annotations: <none>
Selector: app=bose-einstein,tier=front-end
Type: ClusterIP
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.43.67.63
IPs: 10.43.67.63
Port: <unset> 9090/TCP
TargetPort: 8080/TCP
Endpoints:
Session Affinity: None
Internal Traffic Policy: Cluster
Events: <none>
Endpoints is empty: the Service found no Pod. Compare the Selector and TargetPort with what the Deployment really creates:
kubectl describe deployment bose-einstein -n physics
Name: bose-einstein
Namespace: physics
CreationTimestamp: Sat, 10 Oct 2026 04:31:08 +0000
Labels: <none>
Annotations: deployment.kubernetes.io/revision: 1
Selector: app=bose-einstein
Replicas: 2 desired | 2 updated | 2 total | 2 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 25% max unavailable, 25% max surge
Pod Template:
Labels: app=bose-einstein
tier=frontend
Containers:
nginx:
Image: nginx:1.29-alpine
Port: 9090/TCP
Host Port: 0/TCP
Readiness: http-get http://:9090/ delay=0s timeout=1s period=5s successThreshold=1 failureThreshold=3
...
kubectl get pods -n physics --show-labels
NAME READY STATUS RESTARTS AGE LABELS
bose-einstein-846d945d7-kdwjv 1/1 Running 0 44s app=bose-einstein,pod-template-hash=846d945d7,tier=frontend
bose-einstein-846d945d7-lz727 1/1 Running 0 44s app=bose-einstein,pod-template-hash=846d945d7,tier=frontend
| Service has | Pods have | Match? | |
|---|---|---|---|
Label app | bose-einstein | bose-einstein | yes |
Label tier | front-end | frontend | no |
| Port | targetPort: 8080 | container port 9090 | no |
The Deployment Selector is only app=bose-einstein, which the Pods have, so the Deployment is happy. The Service also asks for tier=front-end, which no Pod has, so the Service finds nothing. On top of that, even a correct selector would send traffic to port 8080, where nginx does not listen.
These are the two fixed values from the table above, and both must be corrected in the Service.
Do not "fix" this by changing the Pod label to front-end or the container port to 8080. Both edit the Deployment, which the task does not allow. Do not delete the tier label from the Service selector either: the Service selects on both labels, and both must match the Pods.
Step 4: Fix the Service
cat /home/laborant/bose-einstein/service.yaml
apiVersion: v1
kind: Service
metadata:
name: bose-einstein
namespace: physics
spec:
type: ClusterIP
selector:
app: bose-einstein
tier: front-end
ports:
- port: 9090
targetPort: 8080
Change tier: front-end to tier: frontend, and targetPort: 8080 to targetPort: 9090:
vi /home/laborant/bose-einstein/service.yaml
spec:
selector:
app: bose-einstein
tier: frontend # same as the Pod label
ports:
- port: 9090
targetPort: 9090 # the port nginx listens on
kubectl apply -f /home/laborant/bose-einstein/service.yaml
service/bose-einstein configured
kubectl describe service bose-einstein -n physics | grep -E "Selector|TargetPort|Endpoints"
Selector: app=bose-einstein,tier=frontend
TargetPort: 9090/TCP
Endpoints: 10.42.3.2:9090,10.42.1.2:9090
The Service now finds both Pods, on port 9090. The site still returns 404, because the Ingress still points at port 80:
curl -s http://bose.physics.local/
404 page not found
Step 5: Fix the Ingress
cat /home/laborant/bose-einstein/ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: bose-einstein
namespace: physics
spec:
ingressClassName: traefik
rules:
- host: bose.physics.local
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: bose-einstein
port:
number: 80
Point the Ingress at the port the Service listens on, 9090:
vi /home/laborant/bose-einstein/ingress.yaml
backend:
service:
name: bose-einstein
port:
number: 9090 # same as the Service port
kubectl apply -f /home/laborant/bose-einstein/ingress.yaml
ingress.networking.k8s.io/bose-einstein configured
kubectl describe ingress bose-einstein -n physics
Rules:
Host Path Backends
---- ---- --------
bose.physics.local
/ bose-einstein:9090 (10.42.3.2:9090,10.42.1.2:9090)
Annotations: <none>
Events: <none>
The Ingress now shows both Pod addresses behind the Service.
Step 6: Confirm the site works
curl -s -I http://bose.physics.local/ | head -1
HTTP/1.1 200 OK
curl -s http://bose.physics.local/ | grep -o '<title>[^<]*</title>'
<title>Bose-Einstein Theory</title>
The bose.physics.local tab now shows the page.
Other correct solutions
The Service selector and targetPort have only one correct value each. The ports between the Ingress and the Service are free, so there are several valid ways to connect them.
| Solution | Ingress port | Service port | Service targetPort |
|---|---|---|---|
| A. Change the Ingress (shown above) | number: 9090 | 9090 | 9090 |
| B. Change the Service port, keep the Ingress | number: 80 | 80 | 9090 |
| C. Pick any other port for both | number: 8000 | 8000 | 9090 |
| D. Use a port name instead of a number | name: web | 9090, name: web | 9090 |
Solution B keeps ingress.yaml on port 80 and makes the Service listen on 80 instead:
apiVersion: v1
kind: Service
metadata:
name: bose-einstein
namespace: physics
spec:
type: ClusterIP
selector:
app: bose-einstein
tier: frontend
ports:
- port: 80 # the port the Ingress already uses
targetPort: 9090 # the port nginx listens on
kubectl apply -f /home/laborant/bose-einstein/service.yaml
service/bose-einstein configured
kubectl describe service bose-einstein -n physics | grep -E "Selector|Port|Endpoints"
Selector: app=bose-einstein,tier=frontend
Port: <unset> 80/TCP
TargetPort: 9090/TCP
Endpoints: 10.42.1.5:9090,10.42.2.5:9090
kubectl describe ingress bose-einstein -n physics
Rules:
Host Path Backends
---- ---- --------
bose.physics.local
/ bose-einstein:80 (10.42.1.5:9090,10.42.2.5:9090)
Annotations: <none>
Events: <none>
The request comes in on Service port 80 and is forwarded to 9090 in the Pods.
Solution D gives the Service port a name and refers to it by name in the Ingress:
# service.yaml
ports:
- name: web
port: 9090
targetPort: 9090
# ingress.yaml
port:
name: web
All of these pass the checks. In every one of them, the Service selector is app: bose-einstein plus tier: frontend, and the targetPort is 9090.
Fixing the files in a different order changes the error you see along the way. With only the Ingress fixed, the route exists but the Service has no endpoints:
HTTP/1.1 503 Service Unavailable
no available server
With the Ingress and the selector fixed but targetPort still 8080, the Service has endpoints on the wrong port:
Endpoints: 10.42.1.2:8080,10.42.3.2:8080
HTTP/1.1 502 Bad Gateway
Bad Gateway
In this lab, watching the code go from 404 to 503 to 502 to 200 tells you which link you fixed at each step.
Do not delete and recreate the Deployment, even from the same deployment.yaml. A recreated Deployment gets a new identity (UID), and the check reports it as recreated even when the site works.