Klustered: Level Three

A Deployment with no pods means its ReplicaSet is trying to create them and being refused. The pod never reaches the scheduler, so there's nothing to kubectl logs or describe pod; the story lives one level up, in the ReplicaSet's own events.

kubectl get deploy,rs -l app=klustered
kubectl describe rs -l app=klustered

Look for a FailedCreate event (and a ReplicaFailure condition on the Deployment). It quotes the exact reason the API server rejected the pod, read that message closely before opening the next hint.

The rejection message is the whole clue:

Pod "klustered-..." is invalid: spec.containers[0].resources.requests:
  Invalid value: "128Mi": must be less than or equal to memory limit of 1Mi

The pod's memory request (128Mi, straight from the Deployment template) is larger than its limit (1Mi), and a request can never exceed its limit. But the template's limit is 256Mi, nobody wrote 1Mi. So something rewrote the pod at admission time, as it was being created.

That something is a MutatingAdmissionPolicy: it lets the API server rewrite incoming objects using CEL, no webhook server required. Find it with:

kubectl get mutatingadmissionpolicies,mutatingadmissionpolicybindings

This cluster has one called resource-enforcer bound to the default namespace; on every pod CREATE it overwrites each container's memory limit to 1Mi while leaving the request untouched. The Deployment template is innocent, the mutation hits the pod, not the template, which is why every pod the ReplicaSet submits is rejected and you never see even a Pending one.

Remove the binding first, then the policy:

kubectl delete mutatingadmissionpolicybinding resource-enforcer-binding
kubectl delete mutatingadmissionpolicy resource-enforcer

The ReplicaSet has been retrying on a loop this whole time, so once the policy is gone its next attempt succeeds on its own, no pod to delete, there was never one to begin with. Within a few seconds a pod appears, keeps its real 256Mi limit, and reaches Running:

kubectl get pod -l app=klustered -o jsonpath='{.items[0].spec.containers[0].resources}{"\n"}'

(Impatient? kubectl rollout restart deploy klustered nudges the controller.) Now try curl http://localhost:30000 again, it still won't answer. On to the next layer.

Confirm the routing first, then start questioning the port:

kubectl get svc klustered -o yaml
kubectl get endpoints klustered
curl -v http://localhost:30000     # "Connection refused" = reached the pod, no listener on that port

The Service forwards to targetPort: 666, which matches the Deployment's containerPort: 666. That agreement looks reassuring, but ask yourself whether containerPort actually forces the process to bind 666.

A Service has three port fields, and conflating them is the whole trap:

• port: the port the Service itself is reachable on.
• targetPort: the port on the pod that traffic is forwarded to.
• nodePort: the port exposed on every node (for type: NodePort).

containerPort in the Deployment is documentation only. The kubelet does not enforce it, and the process is free to bind any port it likes. Here the binary listens on a random high port, while containerPort and targetPort were both left at 666, so kube-proxy faithfully forwards 30000 -> pod:666, a closed door. Nothing in kubectl get or describe will tell you the real port; you have to inspect the live socket.

You can't exec into a scratch image, but you can attach an ephemeral debug container that shares the target's network namespace, then run socket tools from there:

POD=$(kubectl get pod -l app=klustered -o jsonpath='{.items[0].metadata.name}')
kubectl debug -it ${POD} --image=nicolaka/netshoot --target=klustered -- ss -tlnp

ss shows both IPv4 and IPv6 listeners in one view, so the LISTEN row reveals the real port (it won't be 666). If the debug image can't be pulled, fall back to the node-side procfs method in the next hint, which needs no tools at all.

Every listening socket is visible from the node in procfs, no tools required in the container. First find the container's host PID, then read its per-namespace socket tables:

CID=$(crictl ps --name klustered -q)
PID=$(crictl inspect "$CID" | jq -r .info.pid)
awk '$4=="0A"{print $2}' /proc/$PID/net/tcp     # state 0A = LISTEN; HEXIP:HEXPORT

Run that and you'll likely get nothing. That is the trap, and the most important lesson in this step:

A Go server started with http.ListenAndServe(":PORT", ...) binds the IPv6 wildcard [::], not 0.0.0.0. The socket is AF_INET6, so it appears in /proc/$PID/net/tcp6, not /proc/$PID/net/tcp. Look at only tcp and you'll wrongly conclude nothing is listening.

Always check both tables (and udp/udp6 for UDP services). This reads both and decodes the hex port to decimal:

for proto in tcp tcp6; do
  echo "== $proto =="
  awk '$4=="0A"{split($2,a,":"); print a[2]}' /proc/$PID/net/$proto \
    | while read -r hex; do printf '  listening on %d\n' "0x$hex"; done
done

tcp prints nothing; tcp6 prints the real port. Decoding by hand: in 00000000000000000000000000000000:1F90, the [::] address is all zeros and 1F90 hex = 8080. That decimal value is your targetPort.

Point the Service's targetPort at the port you discovered (replace <PORT>):

kubectl patch svc klustered --type merge \
  -p '{"spec":{"ports":[{"port":666,"targetPort":<PORT>,"nodePort":30000}]}}'

Or kubectl edit svc klustered and set spec.ports[0].targetPort by hand. Confirm traffic now reaches the app:

curl -v http://localhost:30000

This time you'll get a reply from the app, but it isn't the page you wanted. One layer left.

The data lives on a hostPath-backed PersistentVolume, and the file is definitely on disk, so this isn't a missing-file problem. Two things decide whether a read succeeds: the UID the process runs as, and the ownership/mode of the file. Look at both.

The identity comes from the container's securityContext:

kubectl get deploy klustered -o jsonpath='{.spec.template.spec.containers[0].securityContext}{"\n"}'
# {"runAsGroup":1000,"runAsUser":1000}

The ownership comes from the backing hostPath on the node:

ls -ln /var/data/klustered
# drwx------ ... 0 0 ... index.html   (owner UID 0, mode 700)

A process running as UID 1000 against a directory owned by UID 0 with mode 700. That can't end well.

runAsUser: 1000 makes the container process run as UID 1000. Unlike user namespaces, there is no UID remapping: a hostPath volume keeps whatever ownership it has on the node's disk, and the kernel checks the container's UID against those on-disk owners directly.

The data dir is root:root with mode 700 (owner-only rwx). UID 1000 is not the owner and not in the group, so it can't even traverse the directory, let alone read index.html. The app returns "Failed to read data".

This is one of the most common real-world Kubernetes failures: an image that assumed root, dropped onto a cluster that forces non-root (PodSecurity restricted, OpenShift's arbitrary UIDs, or a hardened securityContext), suddenly unable to touch its own volume.

The production-correct fix is to reconcile the volume's ownership with the UID the app runs as, using an init container that runs as root and chowns the data before the app starts:

kubectl patch deployment klustered --type json -p '[
  {"op":"add","path":"/spec/template/spec/initContainers","value":[
    {"name":"fix-perms","image":"busybox:1.36",
     "securityContext":{"runAsUser":0},
     "command":["sh","-c","chown -R 1000:1000 /data"],
     "volumeMounts":[{"name":"data","mountPath":"/data"}]}
  ]}
]'

The init container is root (UID 0), so it can chown the node's hostPath; the main container stays non-root and now owns its data. The Deployment rolls a new pod; once it's Running:

curl -s http://localhost:30000

You should now see the page and Visits: 1.

Three more ways out, each teaching something:

• fsGroup looks right but silently fails here. Adding spec.template.spec.securityContext.fsGroup: 1000 tells the kubelet to fix volume group ownership, but that mechanism does not apply to hostPath volumes (only to volume types that support ownership management, like most CSI/emptyDir volumes). The read still fails. A great reminder that fsGroup is not a universal fix.
• Run as root (kubectl patch deployment klustered --type json -p '[{"op":"remove","path":"/spec/template/spec/containers/0/securityContext"}]'). Works, because UID 0 owns the data, but it throws away the non-root posture. Quick, and usually the wrong call.
• Loosen perms on the node (chmod -R 755 /var/data/klustered). Works, but only when you control the node, which in the real world you usually don't.

The init-container chown in Hint 3 is the one that keeps the app non-root and doesn't depend on node access, which is why it's the production answer.