No shell, no exec, no problem

The situation

orders-api is up — Running, 1/1 Ready — but it isn't serving anything. The logs only say it refuses to run:

fatal: config invalid, refusing to serve (retrying)

Which config? Invalid how? You reach for the usual door:

$ kubectl -n kubelings exec -it orders-api -- sh
error: Internal error occurred: ... "sh": executable file not found

There is no sh. No cat, no ls, no anything — the image is distroless: the app binary and its runtime, nothing else. That's great for supply-chain hygiene (M6.6 made you strip images for exactly this reason) and terrible for 2am debugging — until you know about ephemeral containers. kubectl debug grafts a temporary container with real tools into the running pod. With --target, it shares the app's PID namespace, and through /proc/<pid>/root/ you can read the app container's entire filesystem — mounts, config files, all of it. No restart, no image rebuild.

  ┌────────────────────┐  
  │ephemeral container │  
  │                    │  
  └────────────────────┘  
            │             
      kubectl debug       
            │             
            ▼             
     ┌───────────────┐    
     │distroless pod │    
     │               │    
     └───────────────┘    
            │             
    namespaces shared     
            │             
            ▼             
 ┌───────────────────────┐
 │same net, PID, volumes │
 │                       │
 └───────────────────────┘

Your task

1. Attach a debug container to orders-api:

   kubectl -n kubelings debug -it orders-api --image=busybox:1.36 --target=orders-api
   

   Copy to clipboard
2. From inside it, find the app process and inspect the config file it reads (the app's filesystem hangs off /proc/1/root/...).
3. The config lives in a ConfigMap. Fix the bad value, wait for the kubelet to project the update (up to ~1 minute), and confirm the logs flip to config ok — serving orders.

kubectl -n kubelings get pod orders-api -o wide
kubectl -n kubelings logs orders-api --tail=5

ls /proc/1/root/etc/app/
cat /proc/1/root/etc/app/config.json

kubectl -n kubelings edit configmap orders-config

# 1. see for yourself, through the shared PID namespace:
kubectl -n kubelings debug -it orders-api --image=busybox:1.36 --target=orders-api
cat /proc/1/root/etc/app/config.json    # → "mode": "debug"
exit

# 2. fix the source:
kubectl -n kubelings patch configmap orders-config --type=merge \
  -p '{"data":{"config.json":"{\"mode\": \"production\", \"flush_interval\": 5}"}}'

kubectl -n kubelings logs orders-api --tail=3    # → config ok — serving orders
kubectl -n kubelings get pod orders-api -o jsonpath='{.spec.ephemeralContainers[*].name}'