What's Inside Distroless Container Images: Taking a Closer Look

GoogleContainerTools' distroless images are often mentioned as one of the ways to produce small(er), fast(er), and secure(r) containers. But what are these distroless images, really? What's the difference between a container image built from a distroless base and one built FROM scratch? Let's dive in!

Why Do We Need Distroless Images?

Container images built FROM full-blown Linux distributions like debian, ubuntu, or their derivatives (e.g., node:lts or python:3) often come packed with tools and libraries. While comprehensive for day-to-day tasks, most of these components are unnecessary for typical containerized applications at runtime. The result? Larger image sizes and more potential vulnerabilities to manage for no good technical reason.

The desire to create smaller, more secure container images by including only the essentials is natural. The most extreme approach to achieving this minimalism is starting FROM scratch - an empty base image - and adding only the application files and packages truly necessary for your container to function.

However, as discussed in the previous post, this approach may require significant manual effort because by default, scratch-based containers miss:

• System directories like /tmp, /home, or /var.
• CA certificates for secure connections.
• User management files (/etc/passwd, /etc/group).
• Shared libraries for dynamically linked applications.
• Time zone information.

...and possibly more.

That's where distroless images come in! The GoogleContainerTools/distroless project provides prebuilt minimal base images that are as close to scratch as possible but have the necessary system files and folders in place.

The only thing you need to get started with the distroless images is to understand their hierarchy and pick the one that best fits your application's requirements.

Meet the First Distroless Image: gcr.io/distroless/static

A good starting point to become familiar with the GoogleContainerTools distroless collection is the gcr.io/distroless/static image:

docker pull gcr.io/distroless/static
docker images

REPOSITORY                     TAG       IMAGE ID       SIZE
gcr.io/distroless/static       latest    5d7d2b425607   1.99MB

Inspecting its filesystem tells us that:

• It's just ~2MB (which is ~25% of the alpine image size 🤯)
• It has a typical Linux distro directory structure inside.
• The /etc/passwd, /etc/group, and even /etc/nsswitch.conf files are in place.
• Certificates and the time zone information are present as well.
• The image is Debian-based (so, there is a distro in the distroless image after all, but it's stripped down to the bones).
• Last but not least, the licenses seem to be preserved (but I'm not a copyright expert).

And that's it! So, it's 99.99% static assets (well, there is a tzconfig executable). No packages, no package manager, no libc, and 0 CVEs:

trivy image gcr.io/distroless/static

gcr.io/distroless/static (debian 12.9)

Total: 0 (UNKNOWN: 0, LOW: 0, MEDIUM: 0, HIGH: 0, CRITICAL: 0)

Given the presence of the above components, using the gcr.io/distroless/static image as a base could be a simple fix for almost all issues outlined in the previous post on pitfalls of FROM scratch containers:

Practice

Not Every Program Is Statically Linked

A nice side benefit of experimenting with FROM scratch containers is that it helps you understand what a program actually needs to run. For a statically linked executable, this typically includes a few configuration files and a proper root filesystem (rootfs) directory structure. But what about a dynamically linked executable?

To explore this, I'll compile a Go program with CGO enabled and run it on a regular Ubuntu machine to examine the dynamically loaded libraries it requires:

package main

import (
  "fmt"
  "os/user"
)

func main() {
    u, err := user.Current()
    if err != nil {
        panic(err)
    }
    fmt.Println("Hello from", u.Username)
}

# syntax=docker/dockerfile:1
# -=== Builder image ===-
FROM golang:1 AS builder
WORKDIR /app

COPY <<EOF main.go
package main

import (
  "fmt"
  "os/user"
)

func main() {
    u, err := user.Current()
    if err != nil {
        panic(err)
    }
    fmt.Println("Hello from", u.Username)
}
EOF

RUN CGO_ENABLED=1 go build main.go

# -=== Target image ===-
FROM ubuntu
COPY --from=builder /app/main /
CMD ["/main"]

docker build -f ~/Dockerfile.cgo -t go-cgo-ubuntu .
docker run --rm go-cgo-ubuntu

Hello from root

The mighty ldd should do the trick:

docker run --rm go-cgo-ubuntu ldd /main

linux-vdso.so.1 (0x00007ffd9acd5000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb862292000)
/lib64/ld-linux-x86-64.so.2 (0x00007fb8624a8000)

The output looks like a standard set of shared libraries needed for a dynamically linked Linux executable, including libc. But, of course, none of them can be found in the gcr.io/distroless/static image...

Meet the Second Distroless Image: gcr.io/distroless/base

The gcr.io/distroless/static image sounds like a perfect choice for a base image if your program is a statically linked Go binary. But what if you absolutely have to use CGO and the libraries you depend on can't be statically linked (I'm looking at you, glibc)? Or you write things in Rust, or C, or any other compiled language with less perfect support of static builds than in Go?

Meet the gcr.io/distroless/base and gcr.io/distroless/base-nossl images!

docker pull gcr.io/distroless/base
docker pull gcr.io/distroless/base-nossl
docker images

REPOSITORY                     TAG       IMAGE ID       SIZE
gcr.io/distroless/static       latest    5d7d2b425607   1.99MB
gcr.io/distroless/base-nossl   latest    ae4cc24e698d   14.8MB
gcr.io/distroless/base         latest    fab58a7ef52e   20.7MB

Inspecting the filesystem of the smaller gcr.io/distroless/base-nossl image tells us that:

• It's about 7 times bigger than gcr.io/distroless/static (but still just ~15MB).
• It's fully based on the gcr.io/distroless/static image (i.e., includes the rootfs layout, CA certificates, tzdata, etc.).
• The extra layer(s) bring a dozen shared libraries - most notably libc, libresolv, libnss, and libpthread.
• ...and again, no typical full-blown Linux distro fluff.

The slightly larger gcr.io/distroless/base image is fully based on the gcr.io/distroless/base-nossl image and brings an extra shared library - libssl (and its dependencies).

Somewhat expectedly, the gcr.io/distroless/base image has a few CVEs that come from the libc and libssl libraries, but most of the time, it shouldn't be a problem because critical CVEs in these libraries rarely occur and are fixed quickly:

trivy image gcr.io/distroless/base

gcr.io/distroless/base (debian 12.9)

Total: 8 (UNKNOWN: 0, LOW: 7, MEDIUM: 1, HIGH: 0, CRITICAL: 0)

Here is how to adjust the target Go image to make it work with the new distroless base:

...build stage remains unchanged...

# -=== Target image ===-
# Replace the 'FROM scratch' with 'FROM gcr.io/distroless/base-nossl'
FROM gcr.io/distroless/base-nossl

COPY --from=builder /app/main /
CMD ["/main"]

The conclusion?

Practice

Not Every Dynamically Linked Use Case Is the Same

I mentioned Rust in the previous section because it's pretty popular these days. Let's see if it can actually work with the gcr.io/distroless/base image. Here is a simple hello-world program:

fn main() {
  println!("Hello world! (Rust edition)");
}

# syntax=docker/dockerfile:1
# -=== Builder image ===-
FROM rust:1 AS builder
WORKDIR /app

COPY <<EOF Cargo.toml
[package]
name = "hello-world"
version = "0.0.1"
EOF

COPY <<EOF src/main.rs
fn main() {
  println!("Hello world! (Rust edition)");
}
EOF

RUN cargo install --path .

# -=== Target image ===-
FROM gcr.io/distroless/base
COPY --from=builder /usr/local/cargo/bin/hello-world /
CMD ["/hello-world"]

Let's try to run it:

docker build -f ~/Dockerfile.rust -t distroless-base-rust .
docker run --rm distroless-base-rust

/hello-world: error while loading shared libraries:
libgcc_s.so.1: cannot open shared object file:
No such file or directory

Oops! Apparently, the gcr.io/distroless/base image doesn't provide all the needed shared libraries. For some reason, Rust has a runtime dependency on libgcc, and it's not present in the container.

Meet the Third Distroless Image: gcr.io/distroless/cc

Turns out, Rust is not so unique in its requirements. This dependency is so common for dynamically linked binaries that even a special distroless base image was introduced - gcr.io/distroless/cc:

docker pull gcr.io/distroless/cc
docker images

REPOSITORY                     TAG       IMAGE ID       SIZE
gcr.io/distroless/static       latest    5d7d2b425607   1.99MB
gcr.io/distroless/base-nossl   latest    ae4cc24e698d   14.8MB
gcr.io/distroless/base         latest    fab58a7ef52e   20.7MB
gcr.io/distroless/cc           latest    6f09ff5d0af8   23.4MB

Inspecting the filesystem of the gcr.io/distroless/cc image tells us that:

• It's fully based on the gcr.io/distroless/base image.
• The new layer(s) add just ~2MB to the image size.
• The image contains libgcc (and its dependencies) and a few other shared libraries.

Here is how to fix the Rust example using the gcr.io/distroless/cc image:

...build stage remains unchanged...

# -=== Target image ===-
# Replace the 'FROM gcr.io/distroless/base' with 'FROM gcr.io/distroless/cc'
FROM gcr.io/distroless/cc

COPY --from=builder /usr/local/cargo/bin/hello-world /
CMD ["/hello-world"]

The intermediate conclusion should be clear by now:

Distroless Images for Interpreted or VM-Based Languages

Some languages (like Python) require an interpreter for a script to run. Some others (like JavaScript or Java) require a full-blown runtime (like Node.js or JVM). Since the distroless images considered so far lack package managers, adding Python, OpenJDK, or Node.js to them might be problematic (you'd need to learn Bazel to build your own derived distroless image).

Luckily, the distroless project seems to support the most popular runtimes out of the box:

• gcr.io/distroless/java - Java 17 & 21 (at the time of writing)
• gcr.io/distroless/nodejs - Node.js 18 & 20 & 22
• gcr.io/distroless/python3 - Python 3

The Python and Node.js images are built on top of the gcr.io/distroless/cc image, and the Java image is built on top of the smaller gcr.io/distroless/base-nossl variant, all adding extra one-two layers with the corresponding runtime and/or interpreter.

Here is what the final image hierarchy looks like:

Practice

How to Build On Top of the Distroless Images

All distroless images are intentionally built without a shell and a package manager. This makes them more secure for production use, but also means that you can't just RUN something in your Dockerfile, if you base it on a distroless image.

Most typically the RUN command is used to:

• Install extra OS-level packages.
• Install application dependencies.
• Build and/or bundle the application.

If you need to install application dependencies or build the application, you can do it in a separate build stage, basing it on a more developer-friendly image, and then copy the built application to the distroless-based runtime image.

For instance, for a Node.js application, you can do it like this:

FROM node:22 AS build
COPY . /app
WORKDIR /app
RUN npm ci --omit=dev

FROM gcr.io/distroless/nodejs22:nonroot
COPY --from=build /app /app
WORKDIR /app
CMD ["hello.js"]

At the same time, adding an extra OS-level package to the distroless base image can be much more complicated. These images are built with Bazel, and you will need to derive a new image from them adding some extra Bazel rules. Copying package files from the build stage is theoretically possible, too, but the process might be a bit fragile.

Who Uses Distroless Base Images?

I use! Mainly the gcr.io/distroless/static one. It's my favorite replacement for FROM scratch. On a more serious note though, there is a number of prominent users of the distroless images:

• Kubernetes
• Knative
• Kubebuilder
• Tekton
• Teleport
• ko (switched to cgr.dev/chainguard/static)
• Jib (switched to eclipse-temurin)

...and 40K+ matches on GitHub code search for FROM gcr.io/distroless.

Pros, Cons, and Alternatives of Distroless Images

GoogleContainerTools distroless images are small, fast, and more secure. For FROM scratch-like use cases, using gcr.io/distroless/static is a no-brainer, and Node.js, Python, and Java apps should definitely at least consider the distroless runtime images.

The project automatically tracks the upstream Debian releases, and it makes CVE resolution in the distroless images as good as it is in the said distro and in the corresponding language runtime (which is often good enough).

At the same time, adding new OS packages to a distroless base is tricky: changing the base itself requires knowing Bazel (and becoming a fork maintainer?), and adding things later on is complicated by the lack of package managers. The choice of base images is limited by the project maintainers, so if your application doesn't fit, you can't benefit from them.

Distroless images also bring extra operational overhead because debugging production workloads without a shell and a package manager becomes a challenge on its own.

So, my opinion is:

If you're keen on the idea of carefully crafting your images from some minimal base, you may want to take a look at the available alternatives:

• Chainguard Images - Use Wolfi as a minimalistic & secure base image, and with the help of two tools, apko and melange, allow building application-tailored images containing only (mostly?) the necessary dependencies. The main downside is that it's proprietary and costs quite a bit of money 🙈
• Chisel - A somewhat similar idea to the above project, but from Canonical, hence, Ubuntu-based. The project is not very new and has some serious users (Microsoft uses it for their .NET runtime images), but the adoption is still not very high.
• Buildah - It's a powerful tool to build container images that, in particular, allows you to build containers FROM scratch, potentially leveraging the host's build tools to avoid installing them in the container. Here is an example.
• Multi-stage builds - No kidding! You can start FROM scratch or a slim runtime base image and carefully copy over only the needed bits from the build stages.

Still want to have minimalistic container images but don't have time for the above wizardry? Then I have a "wizard in the box" for you:

• minT(oolkit) (formerly DockerSlim) - a CLI tool that allows you to automatically convert a "fat" container image into a "slim" one by doing a runtime analysis of the target container and throwing away the unneeded stuff.

You can read more about the struggle of producing decent container images in 👉 this article of mine.

Conclusion

Distroless images are small, fast, and more secure, but they also require some extra knowledge to use them effectively. One needs to understand the difference between available distroless base images and also be aware of the application-specific requirements to choose the optimal variant. Additionally, extending distroless images with extra OS packages is not straightforward, and the choice of supported language runtimes is limited by the GoogleContainerTools project maintainers.

So, when to use the distroless images? Here is my rule of thumb:

• Every time you want to build an image FROM scratch, you should consider the gcr.io/distroless/static image as a better alternative.
• Find yourself adding shared libraries to a FROM scratch image? Try the gcr.io/distroless/base or gcr.io/distroless/cc images instead.
• Running in a highly regulated environment and the security and compliance is top priority? Then the gcr.io/distroless/{java,nodejs,python} images might be worth the try.
• Need to install extra OS-level packages and learning Bazel sounds too burdensome? Check out the alternatives: a multi-stage build with a slim runtime base image, Chainguard, Chisel, and minT(oolkit).

Happy building!