Move Kubernetes Secret Encryption From a Local Key to an OpenBao KMS
Scenario
The Secrets of your kubeadm cluster are encrypted at rest with the aescbc provider.
A new security audit found a problem with the key.
It is stored in plaintext in /etc/kubernetes/enc/encryption-config.yaml, on the same node as etcd.
Anyone who gets a copy of the node's disk gets both the encrypted data and the key that decrypts it.
The security team wants the key out of the cluster.
They run OpenBao, an open-source secrets manager and a fork of HashiCorp Vault.
OpenBao has a transit engine that encrypts and decrypts data with keys that never leave OpenBao.
The team created a transit key called kubernetes for the cluster.
The Kubernetes API server cannot talk to OpenBao on its own.
It talks to a KMS plugin over a Unix socket, and the plugin forwards the requests to OpenBao.
The team has already started the plugin on cplane-01:
| Component | Where |
|---|---|
| OpenBao | http://127.0.0.1:8200, the openbao systemd service |
| KMS plugin | /var/run/kms/openbao.sock, the openbao-kms-plugin systemd service |
You do not need to configure OpenBao or the plugin.
Your shell on cplane-01 has a read-only OpenBao token, so you can look at the key:
bao read transit/keys/kubernetes
The token and the OpenBao address come from /etc/profile.d/openbao.sh.
Login shells load this file, but sudo drops these variables, so run bao without sudo.
If bao fails with server gave HTTP response to HTTPS client, open a new terminal tab or run source /etc/profile.d/openbao.sh.
The cluster has three Secrets, and all of them are encrypted with the local aescbc key right now:
| Namespace | Secret | Created by |
|---|---|---|
secure | db-credentials | the app team |
billing | payment-gateway | the app team |
kube-system | bootstrap-token-<random> | kubeadm init |
You can see the provider of each Secret at the start of its value in etcd:
sudo etcdctl get /registry/secrets/secure/db-credentials --print-value-only \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key | head -c 23; echo
The output is k8s:enc:aescbc:v1:key1:.
Task
Move the encryption of Secrets from the local aescbc key to OpenBao.
Use the kms provider with API version v2 and the plugin socket from the table above.
When you are done:
- New Secrets are encrypted with the
kmsprovider. - All three Secrets from the table above are stored in etcd with the
kmsprovider. - The encryption configuration of the running API server no longer contains the
aescbckey. kubectlstill returns the original values of the Secrets.
Do not delete or change any Secret data.
Keep a copy of every file you change, outside the directory you change it in, so that you can go back.
Hint 1: The kms provider
The kms provider goes into the same providers list as aescbc.
For API version v2, it needs a name, an endpoint, and the apiVersion field.
The endpoint is the plugin socket in the unix:// form.
The first provider in the list encrypts new writes. The API server can read data with any provider in the list.
Docs:
- Using a KMS provider for data encryption (Kubernetes)
Hint 2: The API server cannot reach the plugin
The API server runs in a container. It sees only the host paths that are mounted into its static Pod. The directory with the encryption config is already mounted. The directory with the plugin socket is not.
Hint 3: When the API server does not come back
While the API server is down, kubectl cannot tell you why.
List all containers, including the ones that exited, and read the logs of the last kube-apiserver container:
sudo crictl ps -a --name kube-apiserver
sudo crictl logs <container-id>
The plugin has its own logs:
sudo journalctl -u openbao-kms-plugin
Hint 4: The Secrets that already exist
The API server encrypts a Secret when it writes it to etcd.
A Secret that was written with aescbc stays that way until something writes it again.
The docs page about encryption at rest has a section on how to make sure that all Secrets are encrypted.
Hint 5: Removing the local key
Remove the key only after every Secret is stored with the kms provider.
If the API server loses the aescbc key while some Secrets are still encrypted with it, it cannot read those Secrets anymore.
The API server reads the encryption config when it starts. The kubelet restarts the API server when its manifest changes, but a change of the encryption config alone does not change the manifest.