Send a Signal to a Kubernetes App: a Non-Root Container Case

If you solved the Send a Signal to a Kubernetes App: a Slim Container Case challenge, your first thought might be to use the kubectl debug command. This is definitely a good way to start, but don't be surprised if it doesn't work out of the box.

The kubectl debug command can fail with the following error:

Error from server (Forbidden): pods "non-root" is forbidden:
violates PodSecurity "baseline:latest":
non-default capabilities (container "debugger-pncnl" must not
include "SYS_PTRACE" in securityContext.capabilities.add)

This happens because the default debug profile of kubectl debug adds the SYS_PTRACE Linux capability to every ephemeral container.

The SYS_PTRACE capability is useful for inspecting processes, but it is not allowed by the baseline Pod Security profile enforced in this namespace. As a result, the API server rejects the ephemeral debug container before it is created.

To get past this, look for a kubectl debug --profile option that lets you choose a Pod-Security-compatible debug profile instead of the default one.

The kubectl debug command shouts "container has runAsNonRoot and image will run as root" and then hangs indefinitely? The debugging image runs as root by default, and the Pod's security context does not allow it. You'll need to start the debug container as a non-root user.

Besides --profile, the kubectl debug command has the --custom flag (stable since Kubernetes v1.32). It takes a partial container spec from a JSON or YAML file and applies it to the debug container. Such a spec can include a securityContext with the user that the debug container runs as.

Made the kubectl debug command work but now getting the "can't kill pid 1: Operation not permitted" error? Make sure the UID of the debugging container matches the UID of the target container. 😉 The ps command in the debugging container shows which UID the target process runs as.

You can also start ephemeral containers without kubectl debug. Under the hood, it PATCH-es the Pod's ephemeralcontainers subresource, and a request of your own can set any field of the ephemeral container spec, including its security context.

Here is an example of how to start an ephemeral container called debug-123 attached to the app container and running as a non-root user:

curl -Lvk localhost:8001/api/v1/namespaces/default/pods/non-root/ephemeralcontainers \
  -XPATCH \
  -H 'Content-Type: application/strategic-merge-patch+json' \
  -d '
{
    "spec":
    {
        "ephemeralContainers":
        [
            {
                "name": "debug-123",
                "command": ["sh"],
                "targetContainerName": "app",
                "image": "alpine",
                "stdin": true,
                "tty": true,
                "securityContext": {"runAsUser": 1000}
            }
        ]
    }
}'

After you start such an ephemeral container, you'll still need to find a way to execute the kill command in it, but it shouldn't be too hard.

If you don't feel like writing container spec files or PATCH-ing the Pod's spec manually, you can try another debugging tool called cdebug. It also relies on the ephemeral containers API, and it can set the user of the debug container with a single flag. Most of the playgrounds have it preinstalled: cdebug exec --help.