What is Dagger?

This course is a practical introduction to Dagger - exciting new technology that brings joy back to CI/CD and adjacent domains. The goal is to learn how to use the tool by gradually migrating an example software project from traditional script-based automation to a Dagger-powered development workflow.

However, before we dive into our first hands-on lesson, we should try to understand the essence of Dagger, and drawing parallels with other, perhaps more well-known, tools can help us with that. Doing so will also better prepare us to grasp the rest of the course materials.

What is Dagger? πŸ€” Is it a complete GitHub Actions or Jenkins replacement? Or is it a dev tool that augments the existing CI/CD providers? Is it a service or a CLI? Can it be self-hosted and what's the deal with Dagger Cloud?

To answer these questions, let's see how Dagger can be used in a typical software project and which traditional tools and/or services it can augment or replace.

Setting up the stage

The backbone of this course is a simple yet realistic software project, iximiuz/webgopher, a toy web service written in Go. Its source code is conveniently checked out in the playground as ~/webgopher - just explore it, and you'll quickly become familiar with it.

In almost any modern software project, a few common tasks typically arise:

  • Linting the code
  • Running the tests
  • Building the project
  • Publishing images
  • Etc.

While these tasks are often associated with CI/CD, they are actually part of the broader development workflow. For instance, developers might want to lint the code or run unit tests locally before pushing their changes and triggering the first CI pipeline. Automation of these typical development workflows will be the key theme of the entire course.

First, let's see how these development tasks are codified in Webgopher without Dagger.

Development Workflow Before Dagger

Similar to many other projects, Webgopher relies on Make β€” a build automation tool and task runner β€” to organize its development workflow.

Here is how the linting, testing, and building steps are defined in the project's Makefile:

cat ~/webgopher/Makefile
.PHONY: lint
lint:
  golangci-lint run

.PHONY: test
test:
  go test ./...

.PHONY: build
build:
  CGO_ENABLED=0 go build -o webgopher

πŸ’‘ Despite being half a century old, Make remains extremely ubiquitous. It was originally designed as a build tool but nowadays it's often (ab)used as a general purpose task runner, with PHONY targets used to essentially define tasks.

With all workflow tasks aggregated in the Makefile, developers can handily run them locally:

make lint

# or
make test

# or
make build

...while the CI/CD pipelines can invoke the same make targets on build servers, seemingly unifying the local and remote executions:

# .github/workflows/lint.yml
...
jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-go@v2
      - run: make lint  # <-- the same `make lint` as on the developer's machine

Now let's see where Dagger can come into play.

What if my workflow tasks are (much) more complex than one-liners above?

Over time, the development workflow tasks tend to become more complex. When a Make target becomes too lengthy, it can be refactored into a separate shell script that is then invoked by the original target. Thus, the following project structure is also quite common:

ls my-advanced-project/hack
lint.sh test.sh build.sh
cat my-advanced-project/Makefile
.PHONY: lint
lint:
  ./hack/lint.sh

.PHONY: test
test:
  ./hack/test.sh

.PHONY: build
build:
  ./hack/build.sh

Using more modern task runners like Just or Task instead of Make can help further reduce the complexity. They often come with nice extra features and are devoid of the most annoying Make's peculiarities. However, they do not substantially change the situation, as they are subject to the same design limitations as Make. In the end, such a task runner is just a local process that executes scripts (or ad hoc commands) directly on its host machine.

Development Workflow With Dagger

Getting straight to the point, Dagger replaces traditional task runners (e.g., Make, Just, Task) and the (often messy) scripts they execute. No more, no less.

Dagger itself is a joint of the open source Dagger CLI and Dagger Engine, and an optional proprietary component called Dagger Cloud (SaaS).

Dagger high-level architecture

The only prerequisite for using Dagger is to download the Dagger CLI:

curl -L https://dl.dagger.io/dagger/install.sh | BIN_DIR=/usr/local/bin sh

The initialization of the Dagger Engine happens automatically when you run the Dagger CLI for the first time. The engine is conveniently distributed as a container image, and the Dagger CLI relies on the local docker CLI to pull and launch it.

Using Dagger with alternative container runtimes πŸ‘¨β€πŸ”¬

Dagger can also work with Podman and nerdctl - the only additional thing you need to do is to create an alias like below πŸ‘‡

# For Podman
sudo ln -s $(which podman) /usr/local/bin/docker`

# For nerdctl
sudo ln -s $(which nerdctl) /usr/local/bin/docker`

Remote Dagger Engine execution is also supported, and we'll dig into the details of running Dagger Engine in Kubernetes in one of the future lessons.

Refer to the official documentation for the full list of supported runtimes and integrations.

To start using Dagger in your project, you need to initialize a Dagger module. Like a Makefile, it can reside anywhere within the project's file tree, with the two most common locations being the root of the project or a dedicated subfolder such as ./ci or ./hack (or whatever name you might prefer for it).

cd ~/webgopher

dagger init --sdk=go  # or --sdk=python|typescript

Once the module is initialized, the natural next step is to start adding functions to it. You can think of Dagger functions as PHONY make targets (or Just's recipes, or Task's tasks, no pun intended). But unlike Make targets, Dagger functions are typically written in a full-fledged programming language: Go, Python, TypeScript, or one of the community-maintained SDKs.

⚠️  It's a good approximation to get started, but in actuality, Dagger's functions are much more powerful than Make targets, so don't let this simplification limit your future use of them.

Here is what Webgopher's Dagger module might look like:

~/webgopher/dagger/main.go
package main

type Webgopher struct{}

func (m *Webgopher) Lint() {
  // Lint logic goes here...
}

func (m *Webgopher) Test() {
  // Test logic goes here...
}

func (m *Webgopher) Build() {
  // Build logic goes here...
}

Continuing the analogy with task runners, Dagger functions can be invoked from the command line using the dagger call command:

dagger call lint  # formerly, make lint

# or
dagger call test  # formerly, make test

# or
dagger call build  # formerly, make build

...and functions can also call other functions, but so do Make targets. So, if Dagger is yet another task runner, why anyone would want to use it instead of their current task runner of choice?

The answer is that the way Dagger executes functions is quite different from how traditional task runners execute scripts or commands.

Let's see what's so special about Dagger execution model, and why it's a good idea to start using Dagger in your projects.

Why use Dagger?

The key difference between Dagger and other task runners is that Dagger executes every function in a dedicated container. This may sound odd at first, but it's actually a very clever design decision.

While traditional task runners (potentially combined with scripts) ensure every engineer and CI server run exactly the same commands, Dagger goes one step further and ensures that these commands are executed in identical environments ❀️‍πŸ”₯

Containerized execution of tasks solves one of the fundamental problems of development workflow automation: [the lack of] reproducibility across different machines. Dirty local caches, tool versions' drift, Linux vs. macOS discrepancies stop being issues if you run your tasks with Dagger.

An interesting side-effect of such design is that function call results can be efficiently cached. It makes the subsequent executions of the same function much faster when used locally, but it might be even more important when used in ephemeral CI/CD environments. Dagger supports external cache storage (via Dagger Cloud, but a pluggable implementation is also promised) so you can easily share the cache between different CI/CD runs, too.

πŸ’‘ By their nature, development workflow automation tasks usually modify the state of the local filesystem. When every task execution gets its own isolated fs (thanks to its container), and the call arguments are explicit (thanks to Dagger's programming model which will be covered later), the function call results can be easily cached by snapshotting the container's rootfs.

Another important design decision is that Dagger is a "just a tool".

Dagger cannot replace CI/CD service providers like GitHub Actions, GitLab CI, or Jenkins - someone still needs to supply the compute resources and manage webhooks triggered by project events. But because Dagger is a tool and not a service, you can start using it locally and in CI/CD without major migrations of the existing workflows.

πŸ’‘ Since you only need a statically linked CLI and Docker or the like runtime, all mainstream CI/CD providers should support Dagger "out of the box" (because they usually can run containers). And exactly the same Dagger-powered pipelines can be executed on every development machine, too.

Last but not least, Dagger's programming model also stands out. Dagger functions can call other functions, potentially written in different languages, by other teams, or even imported from third-party repositories. Finally, you can write a build script once and use it in multiple projects without having to copy-paste the code every time.

These Dagger features combined have a potential to really change the status quo in the development workflow automation domain πŸš€

Summarizing

Dagger is a single-host automation engine that takes place of the traditional task runner like Make, Just, or Task and allows writing tasks in real programming languages. It is distributed as a single statically linked binary (CLI), which upon the first run, pulls and launches a container with Dagger Engine (runner). Additionally, Dagger can be augmented with Dagger Cloud - a hosted service that helps operating Dagger-powered CI/CD pipelines.

Dagger truly stands out due to its:

  • Highly parallel, container-native, and portable execution model.
  • The ability to write tasks (functions) in real programming languages.
  • The ability to import and reuse functions written by others (see Daggerverse).
  • Efficient on-disk local caching of function calls.
  • External cache storage and extra pipeline observability (optional, via Dagger Cloud).

What Docker did for application development, Dagger aims to do for development of workflow automation in general and CI/CD in particular πŸš€

Intrigued? In the next lesson, we'll write our first useful Dagger function, learn how to call it from the CLI, and explore what happens from the user's perspective and under the hood. But first, let's practice what we've learned so far.

Practice

It's practice time!

Loading challenge...

Loading challenge...