Challenge ·Medium

Issue Per-Pod mTLS Certificates with PodCertificateRequest

Shiva Swaroop N K
by  Shiva Swaroop N K · on
KubernetesSecurity
Kubernetes 1.37 can hand every pod its own short-lived X.509 identity, no service mesh and no sidecar involved. The API is served out of the box, but it issues nothing without a signer, and a request nobody answers leaves the pod waiting forever. Get a stalled workload its certificates, give its client an identity of its own, and make the server actually enforce mutual TLS.
#mtls#certificates#pki#workload-identity

Your team has been asked to get mutual TLS between two services in the payments namespace. The usual answer is a service mesh: a control plane, a sidecar or a node proxy per workload, and a CA you now operate. For two services, that is a lot of machinery to take on.

Kubernetes 1.37 made an alternative generally available. The kubelet can generate a private key for a pod, ask a signer for a certificate, and drop the result into the pod's filesystem, rotating it before it expires. The pod gets an identity it never has to handle, mint, or renew. No sidecar, no mesh, no bearer token exchanged for a certificate.

The API is served out of the box now. What Kubernetes still does not ship is a certificate authority. It will accept a request, record who is asking, and then wait for something else to answer it. Your platform team put that piece in place: a pod-identity-signer service on cplane-01, with its own CA.

Two workloads are deployed:

  • payments-api: two replicas, one per worker node, meant to serve HTTPS on 8443. Its pods are stuck in ContainerCreating and have never started.
  • checkout: the client. Running, and with no identity of its own.

The plan:

  1. Get payments-api running with certificates the cluster issued it.
  2. Give checkout its own identity.
  3. Make payments-api reject anyone who does not have one.

Constraints:

  • Use the signer that is already installed. Do not replace its CA.
  • Both workloads keep their own service accounts, and their certificates come from the podCertificate projected volume, not from a Secret you fill in yourself.
  • Edit the deployments in place. Do not delete and recreate them.
  • Do not recreate the cluster.

Where this can fail

Three conditions have to hold before a pod holds a certificate it can use, and each one fails in its own way:

  1. The pod asks. A podCertificate projected volume is what makes the kubelet generate a key and file a request at all.
  2. A signer answers. Kubernetes has no built-in CA for this. A request nobody answers stays pending forever, nothing times out on the pod's behalf, and the pod stays in ContainerCreating.
  3. The application uses the certificate for something. A key on disk is not mutual TLS until the server checks the other end.

The first two are platform work. The third is the part a service mesh would have done for you, and the part people forget when they replace one.

Pre-flight

Start on cplane-01:

kubectl -n payments get pods -o wide
kubectl -n payments logs -l app=payments-api --tail=5
journalctl -u pod-identity-signer -n 20 --no-pager

diff <(kubectl -n payments get deployment payments-api -o yaml) /home/laborant/payments-api.yaml

Read all four before you change anything. The last one is the interesting one.

Step 1: Get the server its certificates

The payments-api pods have never started. A pod with a podCertificate projected volume does not start until every certificate in it has been issued, so this is not a crash loop, it is a wait.

kubectl -n payments get podcertificaterequests -o wide

A request carrying no condition has not been answered by anybody. Nothing times out on the pod's behalf, so this state lasts indefinitely.

Hint 1: who is supposed to answer

Kubernetes records the request and stops there. It ships no certificate authority for pod certificates and will never issue one itself. Something has to watch for these requests and write a certificate back into the status.

That something runs on cplane-01, and it reports what it is doing when it starts:

systemctl status pod-identity-signer
journalctl -u pod-identity-signer -n 20 --no-pager

It is up, and it is healthy. So "is the signer running" is not the question.

Hint 2: two names that have to agree

A signer watches for one signer name and ignores every request that does not carry it. A pod asks for one signer name and waits for exactly that one. Put the two side by side:

kubectl -n payments get podcertificaterequests -o wide
journalctl -u pod-identity-signer -n 20 --no-pager | head -3

If they disagree, both halves are working perfectly and neither is talking to the other. The signer takes its copy from its unit:

systemctl cat pod-identity-signer

Either side can be brought into line with the other, and this step is judged on the outcome, so either fix passes.

Hint 3: making a unit change take effect

systemd reads a unit file when it loads it, not when the service restarts. After editing a unit, systemctl restart on its own still runs what systemd loaded earlier. systemctl cat shows you what it currently believes the unit says.

Step 2: Give the client an identity

checkout can already reach the server and verify it, because the CA bundle is mounted at /etc/pod-identity/ca.crt:

POD=$(kubectl -n payments get pod -l app=checkout -o name | head -1)
kubectl -n payments exec ${POD#pod/} -- \
  curl -sS --cacert /etc/pod-identity/ca.crt https://payments-api.payments.svc:8443/

That is one-way TLS. The client knows who the server is; the server has no idea who the client is. Give checkout a certificate of its own.

Add a podCertificate projected volume to the checkout deployment, mounted at /var/run/pod-identity, with the certificate at tls.crt and the key at tls.key, signed by the same signer payments-api uses.

The shape of the volume

You do not have to invent this. payments-api already has a working one, so read its spec:

kubectl -n payments get deployment payments-api -o yaml

The field is also self-documenting from the cluster:

kubectl explain pod.spec.volumes.projected.sources.podCertificate

That output covers every field, including which key types the kubelet will generate, and a third path option that writes the key and the chain into a single file. Prefer that single file when your application can read it, because two separate files can be read mid-rotation and disagree with each other. nginx needs them separate, which is why the server here does not use it.

The identity comes from the service account

checkout runs as the checkout service account, and the signer here turns that into spiffe://cluster.local/ns/payments/sa/checkout. Read what was issued:

kubectl -n payments get podcertificaterequests -o json \
  | jq -r '.items[] | select(.spec.serviceAccountName == "checkout") | .status.certificateChain' \
  | openssl x509 -noout -ext subjectAltName

Two workloads sharing a service account share an identity. That is a design decision about your service accounts, not about certificates.

Step 3: Make the server enforce it

Both workloads now have certificates. Nothing is checking them.

POD=$(kubectl -n payments get pod -l app=checkout -o name | head -1)
kubectl -n payments exec ${POD#pod/} -- \
  curl -sS --cacert /etc/pod-identity/ca.crt https://payments-api.payments.svc:8443/
# payments-api ok
# client-verify=NONE

A request with no client certificate at all is still served. Make payments-api require one and verify it against the workload CA, so that the same request is refused while a request presenting an issued certificate succeeds.

Where the server's configuration lives
kubectl -n payments get configmap payments-api-nginx -o yaml

The server terminates TLS but asks nothing of the client. Two directives are missing, and they do different jobs. One gives nginx a CA to verify client certificates against; the other makes presenting one mandatory instead of optional. The CA bundle is already mounted into the server pod at /etc/pod-identity/ca.crt.

Both directives are in the ngx_http_ssl_module reference; look for the two whose names begin with ssl_verify and ssl_client.

The change is applied but nothing changed

nginx reads its configuration at startup, and updating a ConfigMap does not restart anything by itself. The projected file inside the running container updates on its own schedule, and the process will not notice either way:

kubectl -n payments rollout restart deployment payments-api
kubectl -n payments rollout status deployment payments-api

Other people's work on the same terrain, worth doing alongside this one:

References

How to Author Challenges on iximiuz Labs

Instead of providing a subpar online editing experience, iximiuz Labs offers a helper CLI tool called labctl, allowing you to use your favorite text editor (or a full-featured IDE) to write content from the comfort of your local machine.

Install labctl CLI

curl -sf https://labs.iximiuz.com/cli/install.sh | sh

This will download and install the latest version of the labctl CLI. You only need to do this once per workstation.

Authorize labctl

labctl auth login

This will open a browser window asking you to authorize labctl to access your account. You need to do it after a fresh install of labctl and repeat it whenever the auth session expires.

Pull challenge content

labctl content pull challenge per-pod-mtls-with-podcertificaterequest-144fe512

This will create a local copy of the challenge content in a directory named per-pod-mtls-with-podcertificaterequest-144fe512. You only need to do this once per challenge.

Stream your changes

labctl content push -fw challenge per-pod-mtls-with-podcertificaterequest-144fe512

Run this command in a separate terminal to continuously upload your changes to the server while editing the challenge in your favorite text editor or IDE.

You can also use labctl to create, list, and delete your content. Learn more about the available commands: labctl content --help

Start challenge