Recap: What is ctr?

A short recap of the previous lesson: ctr is a command-line client shipped as part of the containerd project. If you have containerd running on a machine, the ctr binary will likely also be present there. The ctr client is to containerd what the docker client is to dockerd, and, like containerd itself compared to Docker, ctr is a lower-level tool, which may not be as user-friendly as Docker's CLI. But it's still a good idea to learn it because it's a great way to understand how containers work under the hood.

In this lesson, we'll see how to use ctr for basic (run, list, stop, remove) and advanced (create tasks, attach, exec) container management.

ctr image management commands

🧑‍🎓 Similarly to the previous one, the playground for this lesson is a regular Linux server with Docker pre-installed. Since Docker utilizes containerd under the hood, both the containerd daemon and the ctr client will also be available on the host.

Basic ctr commands

One of the main differences between ctr and docker UX is that the former requires doing more things explicitly and doesn't allow you to take (many) shortcuts. For instance, with docker, you can run a container without explicitly pulling its image first. With ctr though, you'll have to pull the image (specifying its full name, including the registry and the tag parts) and only then invoke the ctr run command.

Compare the de-facto standard docker run nginx with the following ctr equivalent:

ctr run docker.io/library/hello-world:latest hello1

You may want to ctr image pull the image first 😉

ctr image pull docker.io/library/hello-world:latest

Notice that unlike user-friendly docker run that generates a unique ID for every container, with ctr, you must supply a container ID yourself (hello1 in the above example).

The ctr run command resembles the docker run command but it doesn't support all the flags you may be used to. For instance, you won't be able to publish container ports or do something like --restart=always. But it also can do things that docker run can't, can you find some? 😉

Back to basic container operations, you can list existing containers with:

ctr container ls

You can also inspect a container with ctr container info <container-id>:

ctr container info hello1

Finally, you can remove a container with ctr container remove <container-id>. Let's remove the hello1 container we've created earlier:

ctr container remove hello1

Note that you can remove only containers that aren't running.

Containers vs tasks

Interesting that the ctr run command is actually a shortcut! It's a combination of ctr container create and ctr task start. Let's explore this behavior:

# Don't forget to pull the image!
ctr container create docker.io/library/nginx:alpine nginx1

If you list the containers ctr container ls, the output will be similar to the following:

CONTAINER    IMAGE                              RUNTIME
nginx1       docker.io/library/nginx:alpine     io.containerd.runc.v2

However, checking the running processes with pgrep nginx will return nothing:

pgrep nginx
# <empty output>

As you can see, the container is created but no process is running inside it yet.

To make the Nginx container actually run, you'll need to start a task:

ctr task start --detach nginx1

If you list the tasks now:

ctr task ls

...the output should be similar to the following:

TASK      PID      STATUS
nginx1    39928    RUNNING

I like this separation of container and task subcommands because it reflects the often forgotten nature of OCI containers. Despite the common belief, containers aren't processes - containers are isolated and restricted execution environments for processes. So, in containerd, a container seems to be a configuration entity that describes the execution environment, while tasks represent the actual processes running inside of containers.

🤓 Note that at least with ctr it doesn't seem to be possible to have multiple tasks running for the same container simultaneously. You can always stop the running task and then start another one for the same container, but you can't have two tasks running at the same time. In particular, it means that the task management commands we'll see below accept container IDs as arguments, not task IDs.

Attaching to background tasks

The nginx1 task from the previous section runs in the background because the ctr task start command was used with the --detach flag. To see the stdout and stderr of a running task, you can attach to it with ctr task attach <container-id>. Let's try attaching to the nginx1 task:

ctr task attach nginx1

The output should be similar to the following:

...
2023/05/06 18:48:22 [notice] 1#1: using the "epoll" event method
2023/05/06 18:48:22 [notice] 1#1: nginx/1.23.4
2023/05/06 18:48:22 [notice] 1#1: built by gcc 10.2.1 20210110 (Debian 10.2.1-6)
2023/05/06 18:48:22 [notice] 1#1: OS: Linux 5.10.175
2023/05/06 18:48:22 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1024:1024
2023/05/06 18:48:22 [notice] 1#1: start worker processes
2023/05/06 18:48:22 [notice] 1#1: start worker process 29
2023/05/06 18:48:22 [notice] 1#1: start worker process 30

But be careful, the ctr task attach command will also reconnect the stdin stream and start forwarding signals from the controlling terminal to the task processes, so hitting Ctrl+C might kill the task.

Unfortunately, ctr doesn't support the Ctrl+P+Q shortcut to detach from a task - it's solely docker's feature. There is also no ctr task logs, so you can't see the stdout/stderr of a task without attaching to it. Neither can you easily see the logs of a stopped task. It's a lower-level tool, remember? 😉

Executing commands in containers

Much like in Docker, you can execute a command in a running container. Let's revive the nginx1 task and execute a command inside the Nginx container:

ctr task start --detach nginx1

Here's how you can get an interactive shell inside the nginx1 container using ctr task exec:

ctr task exec -t --exec-id shell1 nginx1 sh

When you're done exploring the inside of the container, you can exit the shell ending the shell1 exec session:

You can also execute a single command inside the container without getting an interactive shell. For instance, here's how you can curl the Nginx container from the host:

ctr task exec --exec-id curl1 nginx1 curl 127.0.0.1:80

The output will be the standard Nginx welcome page:

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
...

Sending signals to tasks

It's also possible to send signals to tasks, or rather to the processes running inside the tasks. For instance, here is how you can send a SIGTERM signal to the Nginx process, effectively terminating it:

ctr task kill -s 9 nginx1

Interesting that when all container processes are terminated, the task may still exist:

ctr task ls
TASK      PID     STATUS
nginx1    2756    STOPPED

However, the ctr task ps nginx1 command will show that there are no processes running inside the task.

If your only goal for sending a signal is to terminate the task before removal, there might be a faster way to remove a running task - using the ctr task rm command with the --force flag. In any case, let's clean things up and remove the stopped nginx1 task:

ctr task rm nginx1

Summary

In this lesson, you've learned how to use the ctr command-line tool to deal with containers. Since it's a lower-level tool, it requires you to do many things explicitly and doesn't allow you to take lots of shortcuts. Also, it doesn't support all the features you may be used to from docker such as port publishing, automatic container restart on failure, or browsing container logs. However, while it might be tedious to use ctr for day-to-day container management, it's a great tool to learn how containers work under the hood. The knowledge and skills you've gained playing with ctr may come in handy when you'll need to debug container-related issues directly on a Kubernetes node with no other tools available.

In the next lesson, we'll see how contaiNERD CTL, a much more advanced command-line client for containerd, tries to bring the containerd CLI experience closer to that of Docker. But first, it's time for practice! 🎯

Practice

And now, a series of exercises to help you internalize the ctr container management commands, because learning-by-doing rocks! 😉 Unlike the tasks in the preceding lesson, these challenges come without solutions - you'll have to figure them out yourself. But fear not, there will be hints along the way. Good luck!

Loading challenge...

Loading challenge...

Loading challenge...

The above exercises reveal how containerd (at least partially) takes care of the most basic and must-have aspects of running Docker containers like setting up namespaces and cgroups. Is the same true for the more advanced parts like provisioning container networking? The following exercises will help you find out.

Loading challenge...

Loading challenge...

Commentary (spoiler alert)

It turns out that by default, containerd doesn't do much of container network provisioning. Apparently, the only thing a bare containerd would do is creating an empty network namespace for every container. Thus, for example, a containerized Nginx will have just the loopback interface to listen on.

In the real world, though, you'll rarely find a containerd installation not accompanied by a bunch of Container Network Interface (CNI) plugins. These plugins are responsible for setting up container networking, and they come in all shapes and sizes. For instance, the most typical bridge container network is implemented by the epoynmous bridge plugin. containerd has a built-in support for CNI plugins, and more advanced clients, like nerdctl, leverage it to provide a more Docker-like experience for running containers. We'll talk more about CNI plugins in the next module.