Building Container Images FROM Scratch: 6 Pitfalls That Are Often Overlooked

A while ago, I was debunking a (mine only?) misconception that every container image includes a full-blown Linux distro. Using an empty, a.k.a. scratch base image, I created a container holding just a single file - a tiny Hello World program. And, to my utter (at the time) surprise, when I ran that container, it worked flawlessly! This experiment led me to an eye-opening conclusion: starting your images from debian, ubuntu, or even alpine is not always necessary.

But, as often happens in labs, that test was staged 🙈

To make my point stronger, I deliberately oversimplified the example by using a statically linked executable that did nothing more than print a string of ASCII characters on the screen. It was a clever way to validate my hypothesis, but it was far from representing real-world scenarios.

What would happen if I took the experiment a step further and ran a slightly more complex app? Could this reveal any hidden challenges or limitations of container images built from FROM scratch? This exploration isn't just a curiosity-driven experiment; it also offers valuable insights for production applications running in FROM scratch containers. I keep coming across posts from people claiming significant reductions in image size and vulnerabilities after switching to FROM scratch. But is it really as safe and straightforward as it sounds? Let’s find out!

Preparing The Stage

The article is going to be highly hands-on because learning by doing is ❤️. The only prerequisite to follow the examples is a machine with Docker on it.

To make the experiments reproducible, I'll write the mini-programs right in the (multi-stage) Dockerfiles, shamelessly abusing the heredoc feature. However, to avoid bloating the article, I'll keep most of the Dockerfiles collapsed by default and highlight only the important parts instead. Here is the general idea:

# -=== Builder image ===-
FROM golang:1 AS builder
WORKDIR /app

COPY <<EOF main.go
package main

func main() {
  <...test program goes here...>
}
EOF

RUN CGO_ENABLED=0 go build main.go

# -=== Target image ===-
FROM scratch
COPY --from=builder /app/main /
CMD ["/main"]

While all examples are going to be written in Go, the same pitfalls should be at least partially applicable to programs written in other compiled languages (Rust, C/C++, Zig, etc.) and running in FROM scratch containers.

Pitfall 1: Scratch Containers Miss CA Certificates

Probably the most common use case that won't work out of the box in FROM scratch containers is calling other services over HTTPS. Consider this simple snippet that fetches the iximiuz Labs front page:

resp, err := http.Get("https://labs.iximiuz.com/")
if err != nil {
  panic(err)
}
defer resp.Body.Close()

body, err := io.ReadAll(resp.Body)
if err != nil {
  panic(err)
}
fmt.Println("Response:", string(body))

# syntax=docker/dockerfile:1
# -=== Builder image ===-
FROM golang:1 AS builder
WORKDIR /app

COPY <<EOF main.go
package main

import (
  "fmt"
  "io"
  "net/http"
)

func main() {
  resp, err := http.Get("https://labs.iximiuz.com/")
  if err != nil {
    panic(err)
  }
  defer resp.Body.Close()

  body, err := io.ReadAll(resp.Body)
  if err != nil {
    panic(err)
  }
  fmt.Println("Response:", string(body))
}
EOF

RUN CGO_ENABLED=0 go build main.go

# -=== Target image ===-
FROM scratch
COPY --from=builder /app/main /
CMD ["/main"]

When run in a FROM scratch container, it produces the following error:

docker build -f ~/Dockerfile.pitfall-1 -t scratch-https .
docker run --rm scratch-https

panic: Get "https://labs.iximiuz.com/": tls: failed to verify certificate:
x509: certificate signed by unknown authority

goroutine 1 [running]:
main.main()
        /app/main.go:12 +0x185

The fix is pretty straightforward - put the certificate authority (CA) certs at some predefined path in the target container. For instance, the up-to-date /etc/ssl/certs/ folder can be copied from the builder stage:

...builder stage remains unchanged...

# -=== Target image ===-
FROM scratch

# Copy the CA bundle from the builder stage
COPY --from=builder /etc/ssl/certs/ /etc/ssl/certs/

COPY --from=builder /app/main /
CMD ["/main"]

# syntax=docker/dockerfile:1
# -=== Builder image ===-
FROM golang:1 AS builder
WORKDIR /app

COPY <<EOF main.go
package main

import (
  "fmt"
  "io"
  "net/http"
)

func main() {
  resp, err := http.Get("https://labs.iximiuz.com/")
  if err != nil {
    panic(err)
  }
  defer resp.Body.Close()

  body, err := io.ReadAll(resp.Body)
  if err != nil {
    panic(err)
  }
  fmt.Println("Response:", string(body))
}
EOF

RUN CGO_ENABLED=0 go build main.go

# -=== Target image ===-
FROM scratch

# Copy the CA bundle from the builder stage
COPY --from=builder /etc/ssl/certs/ /etc/ssl/certs/

COPY --from=builder /app/main /
CMD ["/main"]

docker build -f ~/Dockerfile.pitfall-1-fixed -t scratch-https-fixed .
docker run --rm scratch-https-fixed

Response: <html>...</html>

This is probably the most well-known pitfall of using FROM scratch containers. Now, let's move to less obvious ones.

Pitfall 2: Scratch Containers Miss Important Rootfs Folders

It's pretty common for a program to create temporary files and folders:

f, err := os.CreateTemp("", "sample")
if err != nil {
  panic(err)
}

fmt.Println("Temporary file:", f.Name())

# syntax=docker/dockerfile:1
# -=== Builder image ===-
FROM golang:1 AS builder
WORKDIR /app

COPY <<EOF main.go
package main

import (
  "fmt"
  "os"
)

func main() {
  f, err := os.CreateTemp("", "sample")
  if err != nil {
    panic(err)
  }

  fmt.Println("Temporary file:", f.Name())
}
EOF

RUN CGO_ENABLED=0 go build main.go

# -=== Target image ===-
FROM scratch
COPY --from=builder /app/main /
CMD ["/main"]

However, creating a tmp file using the above Go snippet fails if the program runs in a FROM scratch container:

docker build -f ~/Dockerfile.pitfall-2 -t scratch-tmp-file .
docker run --rm scratch-tmp-file

panic: open /tmp/sample386939664: no such file or directory

goroutine 1 [running]:
main.main()
  /app/main.go:11 +0xbc

It happens because the /tmp folder is missing in the target container. But it's not the only important system folder that you wouldn't find there!

For this particular case, the fix is simple - make sure the /tmp folder exists in the running container. There are different ways to achieve it, including mounting a folder on the fly:

docker run --rm --mount 'type=tmpfs,dst=/tmp,tmpfs-mode=1777' scratch-tmp-file

Temporary file: /tmp/sample2333717960

However, manually creating the entire rootfs layout might be tedious and error-prone. For instance, you should not forget about the sticky bit for the /tmp folder - the directory ownership and modes need to be set very carefully 😉

Pitfall 3: Scratch Containers Miss Proper User Management

The next thing I'll try to put into a FROM scratch container is the following Go program printing out information about the current user:

user, err := user.Current()
if err != nil {
  panic(err)
}

fmt.Println("UID:", user.Uid)
fmt.Println("GID:", user.Gid)
fmt.Println("Username:", user.Username)
fmt.Println("Name:", user.Name)
fmt.Println("HomeDir:", user.HomeDir)

# syntax=docker/dockerfile:1
# -=== Builder image ===-
FROM golang:1 AS builder
WORKDIR /app

COPY <<EOF main.go
package main

import (
  "fmt"
  "os/user"
)

func main() {
  user, err := user.Current()
  if err != nil {
    panic(err)
  }

  fmt.Println("UID:", user.Uid)
  fmt.Println("GID:", user.Gid)
  fmt.Println("Username:", user.Username)
  fmt.Println("Name:", user.Name)
  fmt.Println("HomeDir:", user.HomeDir)
}
EOF

RUN CGO_ENABLED=0 go build main.go

# -=== Target image ===-
FROM scratch
COPY --from=builder /app/main /
CMD ["/main"]

Let's try to run it and see if it works:

docker build -f ~/Dockerfile.pitfall-3 -t scratch-current-user .
docker run --rm scratch-current-user

panic: user: Current requires cgo or $USER set in environment

goroutine 1 [running]:
main.main()
  /app/main.go:11 +0x23c

That's a pity! Another failure. But is it even fixable?

The cgo is off the scope here (I intentionally disabled it to avoid the dependency on libc or any other shared libraries), so according to the Go stdlib, the only remaining way to fix the problem is by setting the $USER environment variable:

docker run --rm -e USER=root scratch-current-user

UID: 0
GID: 0
Username: root
Name:
HomeDir: /

Seems to work! But containers shouldn't run as root. Can another user be used?

docker run --rm -e USER=nonroot scratch-current-user

UID: 0
GID: 0
Username: nonroot
Name:
HomeDir: /

Ah, shoot! The nonroot user also has the UID 0. In other words, it's the same root but in disguise. Maybe using the --user flag will help?

docker run --user nonroot --rm scratch-current-user

docker: Error response from daemon:
  unable to find user root:
  no matching entries in passwd file.

Nope. But Docker gave me a good pointer here - is the passwd file even there?!

Placing these two files manually into the end image seems to resolve the issue:

...builder stage remains unchanged...

# -=== Target image ===-
FROM scratch

COPY <<EOF /etc/group
root:x:0:
nonroot:x:65532:
EOF

COPY <<EOF /etc/passwd
root:x:0:0:root:/root:/sbin/nologin
nonroot:x:65532:65532:nonroot:/home/nonroot:/sbin/nologin
EOF

COPY --from=builder /app/main /
CMD ["/main"]

# syntax=docker/dockerfile:1
# -=== Builder image ===-
FROM golang:1 AS builder
WORKDIR /app

COPY <<EOF main.go
package main

import (
  "fmt"
  "os/user"
)

func main() {
  user, err := user.Current()
  if err != nil {
    panic(err)
  }

  fmt.Println("UID:", user.Uid)
  fmt.Println("GID:", user.Gid)
  fmt.Println("Username:", user.Username)
  fmt.Println("Name:", user.Name)
  fmt.Println("HomeDir:", user.HomeDir)
}
EOF

RUN CGO_ENABLED=0 go build main.go

# -=== Target image ===-
FROM scratch

COPY <<EOF /etc/group
root:x:0:
nonroot:x:65532:
EOF

COPY <<EOF /etc/passwd
root:x:0:0:root:/root:/sbin/nologin
nonroot:x:65532:65532:nonroot:/home/nonroot:/sbin/nologin
EOF

COPY --from=builder /app/main /
CMD ["/main"]

Validating the fix:

docker build -f ~/Dockerfile.pitfall-3-fixed -t scratch-current-user-fixed .
docker run --user root --rm scratch-current-user-fixed

UID: 0
GID: 0
Username: root
Name: root
HomeDir: /root

docker run --user nonroot --rm scratch-current-user-fixed

UID: 65532
GID: 65532
Username: nonroot
Name: nonroot
HomeDir: /home/nonroot

As a result, the example program works as expected, but manual user management is not fun 🙃

FROM scratch

USER 65532:65532
ENV USER=nonroot

COPY --from=builder /app/main /
CMD ["/main"]

Pitfall 4: Scratch Images Miss Time Zone Info

What time is it in Amsterdam?

loc, err := time.LoadLocation("Europe/Amsterdam")
if err != nil {
  panic(err)
}

fmt.Println("Now in Amsterdam:", time.Now().In(loc))

# syntax=docker/dockerfile:1
# -=== Builder image ===-
FROM golang:1 AS builder
WORKDIR /app

COPY <<EOF main.go
package main

import (
  "fmt"
  "time"
)

func main() {
  loc, err := time.LoadLocation("Europe/Amsterdam")
  if err != nil {
    panic(err)
  }

  fmt.Println("Now in Amsterdam:", time.Now().In(loc))
}
EOF

RUN CGO_ENABLED=0 go build main.go

# -=== Target image ===-
FROM scratch
COPY --from=builder /app/main /
CMD ["/main"]

Well, the above Go snippet won't tell you the answer if you run it in a FROM scratch container:

docker build -f ~/Dockerfile.pitfall-4 -t scratch-tz .
docker run --rm scratch-tz

panic: unknown time zone Europe/Amsterdam

goroutine 1 [running]:
main.main()
  /app/main.go:11 +0x140

Similarly to the CA certificates, the timezone information is traditionally stored on disk (e.g., at /usr/share/zoneinfo) and then looked up by programs at runtime. Since these files cannot appear magically in the FROM scratch container, someone needs to put them in the image first (or mount them upon container startup).

Again, the fix is easy - just copy /usr/share/zoneinfo from the build stage into the target image:

...builder stage remains unchanged...

# -=== Target image ===-
FROM scratch

# Copy the timezone info from the builder stage
COPY --from=builder /usr/share/zoneinfo/ /usr/share/zoneinfo/

COPY --from=builder /app/main /
CMD ["/main"]

# syntax=docker/dockerfile:1
# -=== Builder image ===-
FROM golang:1 AS builder
WORKDIR /app

COPY <<EOF main.go
package main

import (
  "fmt"
  "time"
)

func main() {
  loc, err := time.LoadLocation("Europe/Amsterdam")
  if err != nil {
    panic(err)
  }

  fmt.Println("Now in Amsterdam:", time.Now().In(loc))
}
EOF

RUN CGO_ENABLED=0 go build main.go

# -=== Target image ===-
FROM scratch

# Copy the timezone info from the builder stage
COPY --from=builder /usr/share/zoneinfo/ /usr/share/zoneinfo/

COPY --from=builder /app/main /
CMD ["/main"]

Validating the fix:

docker build -f ~/Dockerfile.pitfall-4-fixed -t scratch-tz-fixed .
docker run --rm scratch-tz-fixed

Now in Amsterdam: 2025-01-12 10:55:00 +0100 CET

The hardest part is to remember to do it 😉

Pitfall 5: Scratch Containers Miss Shared Libraries

So far, all the example programs were compiled with the CGO_ENABLED=0 flag to avoid depending on any shared libraries. But what if the program needs to use an external C library? For instance, a Go program might store some data in an SQLite database - compiling it with CGO_ENABLED=0 won't be an option. Or what if my program was written in Rust instead of Go - by default, it links dynamically to the libc and other system libraries.

Let's try to compile the program from the Pitfall 3: Scratch Containers Miss Proper User Management scenario using the CGO_ENABLED=1 flag:

sed 's/CGO_ENABLED=0/CGO_ENABLED=1/' ~/Dockerfile.pitfall-3 > ~/Dockerfile.pitfall-5

docker build -f ~/Dockerfile.pitfall-5 -t scratch-current-user-cgo .
docker run --rm scratch-current-user-cgo

exec /main: no such file or directory

The above error is a little misleading - the /main file is definitely present in the container image, but what's actually missing is the shared libraries it's linked against.

Here is how you can find out what's missing:

docker cp $(docker create scratch-current-user-cgo):/main .
ldd main

linux-vdso.so.1 (0x00007fffe95e5000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f73f360b000)
/lib64/ld-linux-x86-64.so.2 (0x00007f73f3823000)

The list looks like a standard set of shared libraries pretty much any dynamically linked executable depends on.

To make a dynamically linked executable work in a FROM scratch container, you need to carefully copy the shared libraries in addition to the executable itself:

...builder stage remains unchanged...

# -=== Target image ===-
FROM scratch

# Copy the shared libraries from the builder stage
COPY --from=builder /lib/x86_64-linux-gnu/libc.so* /lib/x86_64-linux-gnu/
COPY --from=builder /lib64/ld-linux-x86-64.so* /lib64/

COPY --from=builder /app/main /
CMD ["/main"]

# syntax=docker/dockerfile:1
# -=== Builder image ===-
FROM golang:1 AS builder
WORKDIR /app

COPY <<EOF main.go
package main

import (
  "fmt"
  "os/user"
)

func main() {
  user, err := user.Current()
  if err != nil {
    panic(err)
  }

  fmt.Println("UID:", user.Uid)
  fmt.Println("GID:", user.Gid)
  fmt.Println("Username:", user.Username)
  fmt.Println("Name:", user.Name)
  fmt.Println("HomeDir:", user.HomeDir)
}
EOF

RUN CGO_ENABLED=1 go build main.go

# -=== Target image ===-
FROM scratch

# Copy the shared libraries from the builder stage
COPY --from=builder /lib/x86_64-linux-gnu/libc.so* /lib/x86_64-linux-gnu/
COPY --from=builder /lib64/ld-linux-x86-64.so* /lib64/

COPY --from=builder /app/main /
CMD ["/main"]

Validating the fix:

docker build -f ~/Dockerfile.pitfall-5-fixed -t scratch-current-user-cgo-fixed .
docker run --rm scratch-current-user-cgo-fixed

panic: user: unknown userid 0

goroutine 1 [running]:
main.main()
        /app/main.go:11 +0x27f

Nice! With the shared libraries in place, the program became... startable.

Pitfall 6: Scratch Containers Miss Network Configuration

This is probably the most devious pitfall of all, but luckily, it's becoming less and less relevant thanks to the changes in glibc version 2.35+ and Go 1.16+.

Consider the following Go program:

resp, err := http.Get("http://example.com")
if err != nil {
  panic(err)
}
defer resp.Body.Close()

body, err := ioutil.ReadAll(resp.Body)
if err != nil {
  panic(err)
}
fmt.Println("Response:", string(body))

# syntax=docker/dockerfile:1
# -=== Builder image ===-
ARG GO_VERSION=1.15
FROM golang:${GO_VERSION} AS builder
WORKDIR /app

COPY <<EOF main.go
package main

import (
  "fmt"
  "io"
  "net/http"
)

func main() {
  resp, err := http.Get("http://example.com")
  if err != nil {
    panic(err)
  }
  defer resp.Body.Close()

  body, err := io.ReadAll(resp.Body)
  if err != nil {
    panic(err)
  }
  fmt.Println("Response:", string(body))
}
EOF

RUN CGO_ENABLED=0 go build main.go

# -=== Target image ===-
FROM scratch
COPY --from=builder /app/main /
CMD ["/main"]

If you compile this program with a Go version before 1.16:

docker build --build-arg GO_VERSION=1.15 \
  -f ~/Dockerfile.pitfall-6 -t scratch-dns-1.15 .

...and run it overriding the DNS host example.com to 127.0.0.1, the program will simply ignore the added /etc/hosts entry and go to the actual example.com host:

docker run --rm --add-host example.com:127.0.0.1 scratch-dns-1.15

Response: <html>...</html>

At the same time, if you compile this program with a Go version 1.16 or later:

docker build --build-arg GO_VERSION=1.16 \
  -f ~/Dockerfile.pitfall-6 -t scratch-dns-1.16 .

...the program will use the /etc/hosts entry and go to 127.0.0.1 instead of example.com (and successfully fail):

docker run --rm --add-host example.com:127.0.0.1 scratch-dns-1.16

panic: Get "http://example.com": dial tcp 127.0.0.1:80: connect: connection refused

goroutine 1 [running]:
main.main()
        /app/main.go:12 +0x1d8

The difference in behavior is caused by Go versions prior to 1.16 trying to copy the behavior of the glibc resolver. Historically, the glibc resolver prioritized DNS over the /etc/hosts file, and since this behavior is rather problematic (due to performance and security reasons), most Linux distros used the /etc/nsswitch.conf file to invert the priorities.

But unlike the /etc/hosts and the /etc/resolv.conf files, that are automatically added to the container by the runtime (e.g., Docker, containerd, etc.), the /etc/nsswitch.conf file would only be present in the container if it was added to the image.

As a workaround, you can add the /etc/nsswitch.conf file with the hosts: files dns line to the container image:

...builder stage remains unchanged...

# -=== Target image ===-
FROM scratch

# Define the resolver priorities explicitly.
COPY <<EOF /etc/nsswitch.conf
hosts: files dns
EOF

COPY --from=builder /app/main /
CMD ["/main"]

# syntax=docker/dockerfile:1
# -=== Builder image ===-
ARG GO_VERSION=1.15
FROM golang:${GO_VERSION} AS builder
WORKDIR /app

COPY <<EOF main.go
package main

import (
  "fmt"
  "io/ioutil"
  "net/http"
)

func main() {
  resp, err := http.Get("http://example.com")
  if err != nil {
    panic(err)
  }
  defer resp.Body.Close()

  body, err := ioutil.ReadAll(resp.Body)
  if err != nil {
    panic(err)
  }
  fmt.Println("Response:", string(body))
}
EOF

RUN CGO_ENABLED=0 go build main.go

# -=== Target image ===-
FROM scratch

# Define the resolver priorities explicitly.
COPY <<EOF /etc/nsswitch.conf
hosts: files dns
EOF

COPY --from=builder /app/main /
CMD ["/main"]

Validating the fix:

docker build --build-arg GO_VERSION=1.15 \
  -f ~/Dockerfile.pitfall-6-fixed -t scratch-dns-fixed .

docker run --rm --add-host example.com:127.0.0.1 scratch-dns-fixed

panic: Get "http://example.com": dial tcp 127.0.0.1:80: connect: connection refused

goroutine 1 [running]:
main.main()
        /app/main.go:12 +0x1d6

The missing /etc/nsswitch.conf file combined with Go resolver behavior was one of the reasons the Kubernetes project decided to switch from scratch to distroless images.

Today, both Go (1.16+) and glibc (2.35+) have changed their behavior to check the /etc/hosts file first by default, but are you absolutely sure that there is no legacy software in your environment? 😉

Was it the last pitfall of using the scratch base image? I'm not sure. But it's definitely enough pitfalls for me to start thinking of an alternative.

Conclusion

Should a container have a full-blown Linux distro inside? The short answer is no. But in reality, it's more nuanced. While FROM scratch containers might seem functional, they often lack essential components that many programs expect to find in their execution environment: CA certificates, a proper root filesystem layout, timezone data, the /etc/nsswitch.conf file, and more. The absence of these files can lead to subtle, hard-to-diagnose bugs. To avoid this, you can either add these elements manually during the build process or opt for an alternative minimal base image, and GoogleContainerTools' gcr.io/distroless/static and gcr.io/distroless/base are two excellent options to consider.