Encrypt Kubernetes Secrets at Rest in etcd

Scenario

By default, Kubernetes stores all object data — including Secrets — as plaintext inside etcd. Anyone with access to etcd's data directory or an etcd backup file can read every Secret value directly, without ever going through the Kubernetes API or RBAC.

A Secret named db-credentials already exists in the secure namespace, and right now its username and password values are sitting in etcd completely unencrypted. Confirm this yourself:

sudo ETCDCTL_API=3 etcdctl get /registry/secrets/secure/db-credentials \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

You will see the raw admin / SuperSecret123 values readable in the output. Your job is to fix this by enabling encryption at rest for Secrets.

Task

1. Create an EncryptionConfiguration file at /etc/kubernetes/enc/encryption-config.yaml that encrypts the secrets resource using a real encryption provider (aescbc, aesgcm, or secretbox) — not just identity
2. Update the kube-apiserver static pod manifest to use this configuration via --encryption-provider-config, and mount the config directory into the pod
3. Recreate the existing db-credentials Secret so it picks up the new encryption — encryption only applies to writes that happen after it's enabled, so the Secret must be rewritten to actually become encrypted in etcd
4. Confirm db-credentials is now stored encrypted in etcd, and that it's still readable correctly through kubectl get secret — proving the API server can both encrypt and decrypt transparently

sudo mkdir -p /etc/kubernetes/enc
sudo vi /etc/kubernetes/enc/encryption-config.yaml

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <base64-encoded-32-byte-key>
      - identity: {}

head -c 32 /dev/urandom | base64

sudo vi /etc/kubernetes/manifests/kube-apiserver.yaml

- --encryption-provider-config=/etc/kubernetes/enc/encryption-config.yaml

volumeMounts:
- mountPath: /etc/kubernetes/enc
  name: enc
  readOnly: true

volumes:
- hostPath:
    path: /etc/kubernetes/enc
    type: DirectoryOrCreate
  name: enc

kubectl get pod -n kube-system -l component=kube-apiserver -w

kubectl get secret db-credentials -n secure -o json | kubectl replace -f -

sudo ETCDCTL_API=3 etcdctl get /registry/secrets/secure/db-credentials \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

kubectl get secret db-credentials -n secure -o jsonpath='{.data.username}' | base64 -d
kubectl get secret db-credentials -n secure -o jsonpath='{.data.password}' | base64 -d

Test Cases

Understanding the Challenge

By default, Kubernetes stores all object data in etcd as plaintext — including Secrets. Anyone with access to etcd's data directory, an etcd backup file, or even a privileged pod that can read etcd's storage can extract every Secret value directly, completely bypassing the Kubernetes API and any RBAC rules you have in place.

This challenge fixes that by configuring encryption at rest — a feature where the API server encrypts Secret data before writing it to etcd, and transparently decrypts it again whenever it's read back through the API.

How Encryption at Rest Works

The API server is the only component that talks to etcd directly. When encryption at rest is enabled, the API server sits in the middle of every read and write:

kubectl create secret
        |
        v
  kube-apiserver  --[encrypts]-->  etcd (stores ciphertext)
        ^
        |
  kube-apiserver  <--[decrypts]--  etcd (returns ciphertext)
        |
        v
kubectl get secret (plaintext)

Nothing else in the cluster needs to know encryption is happening. kubectl get secret always returns plaintext because the API server decrypts on the way out. The only way to actually see whether encryption is working is to bypass the API server and read the raw key directly from etcd using etcdctl.

A critical detail: enabling encryption only affects future writes. Any Secret that already existed in etcd before encryption was turned on stays in plaintext until it is written again — Kubernetes does not retroactively re-encrypt existing data on its own.

Step 1 : Confirm the Secret Is Currently Plaintext

Before making any changes, prove the problem exists. Read the db-credentials Secret directly from etcd, bypassing the API server entirely:

sudo ETCDCTL_API=3 etcdctl get /registry/secrets/secure/db-credentials \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

/registry/secrets/secure/db-credentials
...stringData...admin...SuperSecret123...

The admin username and SuperSecret123 password are visible in plain text inside etcd's raw storage. This is the exact data exposure that encryption at rest prevents.

Step 2 : Create the EncryptionConfiguration

Create the directory and the configuration file:

sudo mkdir -p /etc/kubernetes/enc

Generate a real encryption key — 32 random bytes, base64-encoded:

head -c 32 /dev/urandom | base64

W4t3rM3l0nR4nd0mK3y1234567890ab==

Create the file:

sudo vi /etc/kubernetes/enc/encryption-config.yaml

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: W4t3rM3l0nR4nd0mK3y1234567890ab==
      - identity: {}

Breaking down the file:

• resources: [secrets] — only Secret objects are affected by this rule. Other resources like ConfigMaps remain stored as plaintext.
• providers is an ordered list. The first provider is used for all new writes. aescbc here means AES encryption in CBC mode — a real, secure encryption provider.
• identity: {} is the no-op provider — it stores data as plaintext. It must be listed last, as a fallback so the API server can still read any Secrets that were written before encryption was enabled. If identity were listed first, nothing would actually get encrypted.

Step 3 : Update the kube-apiserver Manifest

Edit the kube-apiserver static pod manifest:

sudo vi /etc/kubernetes/manifests/kube-apiserver.yaml

Add the flag that points to the encryption config, inside the container's command args:

- --encryption-provider-config=/etc/kubernetes/enc/encryption-config.yaml

The container also needs a volume mount so it can actually read the file — add a volumeMounts entry under the container:

volumeMounts:
- mountPath: /etc/kubernetes/enc
  name: enc
  readOnly: true

And a matching hostPath volume under spec.volumes:

volumes:
- hostPath:
    path: /etc/kubernetes/enc
    type: DirectoryOrCreate
  name: enc

Save the file. The kubelet watches /etc/kubernetes/manifests/ and detects the change automatically — no manual restart needed.

Step 4 : Wait for kube-apiserver to Restart

kubectl get pod -n kube-system -l component=kube-apiserver -w

NAME                       READY   STATUS        AGE
kube-apiserver-cplane-01   1/1     Terminating   30m
kube-apiserver-cplane-01   1/1     Running       5s

kubectl wait --for=condition=Ready pod \
  -l component=kube-apiserver -n kube-system --timeout=120s

pod/kube-apiserver-cplane-01 condition met

Step 5 : Recreate the Existing Secret

Encryption is now active for new writes only. The db-credentials Secret created during init setup is still sitting in etcd as plaintext, because it was written before encryption was turned on. Force it to be rewritten without changing any of its data:

kubectl get secret db-credentials -n secure -o json | kubectl replace -f -

secret/db-credentials replaced

This reads the Secret out (still decrypted normally, since the old data was plaintext) and immediately writes it straight back in — and that write now goes through the active aescbc provider.

Step 6 : Verify the Secret Is Now Encrypted in etcd

Read the same key directly from etcd again:

sudo ETCDCTL_API=3 etcdctl get /registry/secrets/secure/db-credentials \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

/registry/secrets/secure/db-credentials
k8s:enc:aescbc:v1:key1:\x8f\x1a\x3c...binary ciphertext...

The k8s:enc:aescbc:v1:key1: prefix confirms the data is now encrypted — this prefix is added by the API server to every encrypted value so it knows which provider and key to use when decrypting later. The admin / SuperSecret123 values are no longer visible anywhere in the raw output.

Step 7 : Confirm Decryption Still Works Through the API

Encryption must be completely transparent to normal usage. Confirm the actual Secret values still come back correctly through kubectl:

kubectl get secret db-credentials -n secure -o jsonpath='{.data.username}' | base64 -d

admin

kubectl get secret db-credentials -n secure -o jsonpath='{.data.password}' | base64 -d

SuperSecret123

Both values decrypt correctly — the API server transparently encrypts on write and decrypts on read, and the data stored in etcd is now unreadable to anyone without API server access.