Making Sense Out of Native Sidecar Containers in Kubernetes

Kubernetes 1.28 introduced a new type of container - "native" sidecars. Why this addition was needed? How do the new sidecar containers compare to regular and init containers? What use cases does this new type of container enable? And most importantly, how can I get my hands dirty with the hot new feature? Let's find out!

First, a potentially disturbing surprising fact: the 1.28 release didn't add a new Sidecar type or even a sidecarContainers field to the Pod spec - the so-called "native sidecar" containers are just the same old init containers but with some new properties, which make them behave unlike any other type of containers. At first glance, this may seem like a bad design decision, but when you wrap your head around it, you'll see that it's actually a clever and future-proof solution to a rather delicate architectural problem.

Before diving into the details of the new sidecar containers, let's quickly recap the original difference between the regular and init containers in Kubernetes.

Regular vs. init containers

Every Pod must have at least one regular container defined in its .spec.containers list. When a Pod has multiple regular containers, they all start and run concurrently, and if some of the Pod's containers terminate, they become subject to the Pod-wide restart policy:

apiVersion: v1
kind: Pod
metadata:
  name: pod-01
spec:
  containers:
  - name: app
    image: app:latest
  - name: cache
    image: cache:latest
  restartPolicy: Always | OnFailure | Never

Multiple concurrent and restartable containers deployed together as a single "unit" is a powerful abstraction, but there are also situations when some of the containers in such a group may need to:

• Start before others (i.e., the startup order becomes important).
• Run to completion (i.e., restart upon termination is undesirable).

Enforcing the startup order of regular containers is tricky but doable - a bunch of ugly-looking shell scripts usually do the trick. However, running some of the containers to completion while keeping others restartable is a much harder problem to solve. Simply setting the restart policy to OnFailure is not good enough because when one of the regular containers terminates, the Pod stops being ready, meaning that it's no longer able to serve traffic even if its primary container is still up and running.

A proper solution was needed to address the above problems, and that's how the init containers were born.

Since a while ago, the Kubernetes Pod spec has gotten another list - .spec.initContainers:

apiVersion: v1
kind: Pod
metadata:
  name: pod-02
spec:
  initContainers:
  - name: config-loader
    image: config-loader:latest
  containers:
  - name: app
    image: app:latest
  - name: cache
    image: cache:latest
  restartPolicy: Always | OnFailure | Never

Even though the elements of this new list have a literally identical set of attributes to the regular containers, the init containers behave differently.

Init containers:

• Triggered before regular containers.
• Started one by one in the order defined by their list.
• Run to completion before the next init container starts.

...and because of the above design, init containers:

• Can't respect the Pod's restart policy if it's set to Always - init containers then fallback to OnFailure.
• After their (successful) termination, don't affect the Pod's readiness - otherwise, the Pod with init containers would never become ready.
• Don't allow specifying the startup, liveness, and readiness probes - because they would be useless anyway.