Send a Signal to a Kubernetes App: a Non-Root Container Case

In this rather advanced challenge, you'll need to send a signal to an unprivileged container that uses a FROM scratch base image:

Your cluster access is restricted to basic app management tasks.
The app container runs in the Pod called non-root.
You're not allowed to modify or redeploy the Pod.
The Pod's security context disallows containers running as root.
There is no shell in the target container.

Intriguing enough? Then give it a try!

Hint 1

If you solved the Send a Signal to a Kubernetes App: a Slim Container Case challenge, your first thought might be to use the kubectl debug command. This is definitely a good way to start, but don't be surprised if it doesn't work out of the box.

Using "kubectl debug" with the "--target" option to attach to a specific container in a running Pod.

Hint 2

The kubectl debug command can fail with the following error:

Error from server (Forbidden): pods "slim" is forbidden:
violates PodSecurity "baseline:latest":
non-default capabilities (container "debugger-r4d4s" must not
include "SYS_PTRACE" in securityContext.capabilities.add)

This happens because kubectl debug may want to add the SYS_PTRACE Linux capability to the ephemeral container, especially when the debug container targets another container in the same Pod.

The SYS_PTRACE capability is useful for inspecting processes, but it is not allowed by the baseline Pod Security profile enforced in this namespace. As a result, the API server rejects the ephemeral debug container before it is created.

To get past this, look for a kubectl debug --profile option that lets you choose a Pod-Security-compatible debug profile instead of the default one.

Hint 3

The kubectl debug command shouts "container has runAsNonRoot and image will run as root" and then hangs indefinitely? You may want to try a debugging image that doesn't run as root by default.

Hint 4

Don't know where to find such an image? You can always build a debugging image of your own.

docker build -t registry.iximiuz.com/my-debugging-image:v1.0.0 -<<'EOD'
FROM alpine:3
USER 1002
EOD

docker push registry.iximiuz.com/my-debugging-image:v1.0.0

Hint 5

Made the kubectl debug command work but now getting the "can't kill pid 1: Operation not permitted" error? Make sure the UID of the debugging container matches the UID of the target container. 😉

Hint 6

Don't feel like building images? There is a more fine-grained way to start ephemeral containers than kubectl debug - PATCH-ing the Pod's spec directly, which, in particular, allows you to specify the security context of the ephemeral container.

Here is an example of how to start an ephemeral container called debug-123 attached to the app container and running as a non-root user:

curl -Lvk localhost:8001/api/v1/namespaces/default/pods/non-root/ephemeralcontainers \
  -XPATCH \
  -H 'Content-Type: application/strategic-merge-patch+json' \
  -d '
{
    "spec":
    {
        "ephemeralContainers":
        [
            {
                "name": "debug-123",
                "command": ["sh"],
                "targetContainerName": "app",
                "image": "alpine",
                "stdin": true,
                "tty": true,
                "securityContext": {"runAsUser": 1000}
            }
        ]
    }
}'

After you start such an ephemeral container, you'll still need to find a way to execute the kill command in it, but it shouldn't be too hard.

Hint 7

As you saw in the previous hint, ephemeral containers are quite flexible. However, the kubectl debug command doesn't do a good job exposing the full power of the ephemeral containers API. If you don't feel like building custom debugging images or PATCH-ing the Pod's spec manually, you can try another debugging tool called cdebug, which also relies on the ephemeral containers API but hopefully provides a friendlier UX. Most of the playgrounds have it preinstalled: cdebug exec --help.