Docker Run, Attach, and Exec: How They Work Under the Hood (and Why It Matters)

You'll see the docker run command in every Docker tutorial or demo. It looks simple enough - docker run -it debian or even docker run nginx. But that simplicity is deceptive. What actually happens under the hood may surprise you.

The difference between Docker's attach and exec commands is a common source of confusion - and understandably so. Both commands have similar arguments and, at first glance, similar behavior. However, they aren't interchangeable. They're designed for different use cases, and their implementations differ accordingly. Still, it can be tricky to remember when to use which.

So let's break down what run, attach, and exec really do by looking at how they're implemented. This understanding will make it easier to remember the right tool for the job. And, like any real understanding, it frees you from rote memorization and gives you the power to extrapolate the knowledge to other container runtimes - like Podman, nerdctl, containerd, and even Kubernetes 😎

Docker architecture overview

First, a quick recap of the Docker architecture:

Three takeaways that are particularly important for our discussion:

• Container management architecture is layered.
• Containers are regular processes isolated and restricted execution environments for processes
• There is a shim component between the container manager and the container.

What does docker run command really do?

When you docker run a container, it may feel like you're starting a regular foreground process - the command blocks your current terminal session, and the containerized application's stdout and stderr are printed to your terminal. Anything you type in the terminal is sent to the containerized application's stdin, and if you hit Ctrl+C, accidentally or on purpose, the interrupt signal (SIGINT) is sent to the containerized application.

However, in reality, the docker CLI is just a (relatively thin) client of the dockerd daemon, so the docker run process is not even a parent process of the containerized application (and neither is dockerd). See it for yourself:

docker run -it debian

...and list all processes on the machine from another terminal:

ps auxf

To simulate the "foreground process" experience, the docker run command establishes an elaborate client-server relay between your terminal and the containerized application, making it look like the containerized application is the foreground process.

Remember the layered container management architecture from the previous section? Here is what actually happens when you start a container with docker run:

terminal <-> docker <-> dockerd <-> containerd <-> shim <-> application (container)

Containerized application and stdio streams

Inside a container, there is a regular Linux process (or a few of them). However, if the foreground process in your terminal is docker run, the containerized application must always be a background, daemon-like process.

As every normal process, the containerized application has stdio streams: stdin, stdout, and stderr. Back in the day, when you started a process as a daemon (i.e., detaching it from the starter process), it would be reparented to PID 1, and its stdio streams would be simply closed. However, it's clearly not happening with Docker containers, allowing us to see the containerized application's stdout and stderr in our terminal, and even send data to its stdin.

CONT_ID=$(docker run -d nginx)

...and after a while, you can re-attach to Nginx, making it again feel like a foreground process:

docker attach $CONT_ID

Docker came up with a clever idea of putting an extra process between the container and the rest of the system, called a container runtime shim. In the docker run example above, the container manager actually starts a shim process, that in turn, uses an OCI-compatible runtime (e.g., runc) to start the actual container.

This is the shim process that becomes a daemon - it's reparented to PID 1 and its stdio streams get closed. At the same time, the shim takes the full control over the stdio streams of the containerized application. The daemonized shim process reads from the container's stdout and stderr and dumps the read bytes to the log driver.

What does docker attach command really do?

Compared to docker run, the docker attach command is much less frequently used. However, it's still a useful tool to have in your toolbox because most production applications run in background containers, and attach allows you to re-connect to their stdio streams should you need to.

The container runtime shim component actually acts as a server. It provides RPC means (e.g., a UNIX socket) to connect to it. And when you do so, it starts streaming the container's stdout and stderr back to your end of the socket. It can also read from this socket and forward data to the container's stdin. Hence, you re-attach to the container's stdio streams.

Thus, if you start a background container using the docker run -d|--detach command, you can re-attach to its stdio streams later using docker attach <container>.

Here is what happens when you attach to a running container:

terminal <-> docker <-> dockerd <-> containerd <-> shim <-> container's stdio streams

Difference between docker attach and docker logs

On the diagram above, docker attach streams the container's logs back to the terminal. However, the docker logs command does a similar thing. So, what's the difference?

The logs command provides various options to filter the logs while attach in that regard acts as a simple tail. But what's even more important is that the stream established by the logs command is always unidirectional and connected to the container's logs, not the container's stdio streams directly.

docker logs -f $CONT_ID

The logs command simply streams the content of the container's logs back to your terminal, and that's it. So, regardless of how you created your container (interactive or non-interactive, controlled by a pseudo-terminal or not), you cannot accidentally impact the container while using the logs command.

However, when attach is used:

• If a container was created in the interactive mode (-i), everything you type in the terminal after attach-ing to the container will be sent to its stdin.
• You can (intentionally or accidentally) send a signal to the container - for instance, hitting ctrl+c on your end while attached sends SIGINT to the container.

What does docker exec command really do?

It's time to tackle the docker exec command. This command is frequently used in troubleshooting and debugging scenarios, when you need to execute a command or start an interactive shell inside an already running container.

The exec command may resemble the attach command because you're targeting an existing container. However, in the case of attach, we are merely connecting our terminal to the containerized application's stdio streams (and starting to forward signals), while the exec command rather starts a new container, but kinda sorta inside the existing container.

docker exec -it $CONT_ID curl 127.0.0.1:80

In other words, exec is a form of the run command 🤯

The trick here is that the auxiliary container created by the exec command shares all the isolation boundaries of the target container. I.e., the same net, pid, mount, etc. namespaces, same cgroups hierarchy, etc. So, from the outside, it feels like running a command inside an existing container.

The confusion of attach and exec commands arises because the exec-uted command is also a process with its own stdio streams. So, you can choose whether to exec in detached mode, whether to keep the stdin open, whether to allocate a pseudo-TTY, etc. Also, when exec-ing, the relay looks quite similar to the one established by the attach command:

terminal <-> docker-cli <-> dockerd <-> containerd <-> shim <-> command's stdio streams

How do other runtimes implement run, attach, and exec?

The above diagrams and examples were mostly about Docker, but other container managers, such as containerd or nerdctl, behave similarly when it comes to run, exec, or attach commands.

Podman is probably the most prominent example of a daemon-less container manager. However, even Podman employs container runtime shims. There is just one less hop in the relay when you attach to a Podman's container.

An interesting specimen is Kubernetes. Kubernetes doesn't manage containers directly. Instead, every cluster node has a local agent, called kubelet, that in turn expects a CRI-compatible container runtime to be present on the node. However, on the lowest level, there are still the same shims and processes:

Similar to Docker, Kubernetes' command-line client, kubectl, also provides run, exec, and attach commands with an almost identical UX. The difference is that kubectl calls the central Kubernetes API server (instead of going straight to the container runtime or the corresponding kubelet on the target node), and asks it to create a pod, not a container. But luckily, pods are just groups of semi-fused containers, and having the API server on the way only adds an extra hop to the relay sequence, so conceptually, everything we learned so far is applicable to Kubernetes, too.

Kubernetes Container Runtime Interface (CRI): A concise gRPC API a container runtime needs to implement to be compatible with Kubernetes.

Summarizing

So, to summarize:

• Containers are isolated and restricted execution environments.
• Conventionally, there is one main process per container (but it can have a few children processes).
• Containers are typically started in the detached mode (i.e., like daemons).
• Container runtime shim wraps the container process and streams its stdout and stderr to logs.
• Runtime shim allows attach-ing to it to wire the terminal with the container's stdio stream.
• It's possible to start a container reusing the isolation primitives of an already running container.
• The exec command is similar to the run command with all the namespaces and cgroups reused from another container.
• Since exec-ing by default happens in the attached mode, it might look similar to the attach command, but its purpose and implementation are quite different.

Happy containerizing! 🐳