Debug a Go Container Image That Fails to Start After a Recent Change
The ~/app directory on the workstation machine holds a small Go web service and its release tooling.
The service is containerized following the recommended practices:
a multi-stage build compiles the program with CGO_ENABLED=0
and copies the resulting self-contained binary into a gcr.io/distroless/static runtime image.
The final image has no shell, no package manager, and weighs just a few megabytes.
This setup has worked for a long time.
make run builds and starts the service for local development,
and make docker-push builds the production image and pushes it to the registry.
The latest release, however, is broken.
The image still builds without a single warning, the local build still works as before,
but a container started from the new image exits immediately with:
exec /app/server: no such file or directory
Find the cause and fix the image.
Then build it as registry.iximiuz.com/app:v1.0.0 and push it to the playground's registry.
A separate machine pulls the image from the registry and checks that:
- A container started from the image keeps running and responds on port
8080. - The image stays minimal - with no shell inside and under
50 MBin size. - The application is the unmodified one from
~/app, built with the same flags the Makefile uses.
Follow the dynamic hints of the solution checker. They tell you which requirement is not met.
Hint 1
The project's git history is the best place to start: git log in ~/app shows what changed recently.
Hint 2
One of the recent commits changes the command that builds the binary. Read up on what the new flag changes in the produced executable, and keep in mind that the very same command works on the host.
Hint 3
A position-independent executable (PIE) is a program that can be loaded at any address in memory,
not only at the fixed address it was linked for.
This lets the kernel apply address space layout randomization (ASLR) to the program's own code,
which makes memory corruption bugs harder to exploit.
Hardening guides and container image scanners often check for it.
Go supports it with the -buildmode=pie flag.
Hint 4
The error message the container fails with is misleading.
The kernel reports no such file or directory not only when the executable itself is missing,
but also when something the executable needs at load time cannot be found in the surrounding filesystem.
Build the binary on the host with make build and inspect it with file, ldd, and readelf -l.
What do the tools say about linking and about the program interpreter. Do they all agree?
Hint 5
Go's internal linker can produce a position-independent executable (PIE) without any C toolchain,
but the resulting binary still declares a program interpreter (the dynamic loader) in its ELF headers,
even though it has no shared library dependencies (CGO_ENABLED=0).
The kernel then refuses to start the program when the interpreter is not present in the container's root filesystem.
Hint 6
Which loader is requested depends on the C library of the build environment:
/lib/ld-musl-x86_64.so.1 for a binary linked in an Alpine image and /lib64/ld-linux-x86-64.so.2 for one linked on a glibc-based distro like Debian.
So the runtime image needs to provide the loader that matches the build image.
Hint 7
The distroless collection has image variants that contain glibc and its loader:
There is no musl-based variant, though, so a binary linked in the golang:1-alpine image cannot start in any of them.
Other distroless-style image families exist (for instance, Chainguard's),
but most of them don't ship musl variants either.
So changing the build stage's base image to a glibc-based one is rather inevitable to solve this challenge.