CKA Practice: Renew Expiring Control Plane Certificates

About a year ago someone provisioned this three-node cluster with kubeadm. The SRE on call today ran a routine check to confirm all services were up. Every kubectl command failed:

Unable to connect to the server: tls: failed to verify certificate:
x509: certificate has expired or is not yet valid

The cluster is unmanageable, and the application in namespace web is still serving traffic. Both of those are normal, and the reason is worth understanding before you start. Running pods do not depend on the control plane to keep running. Their containers are supervised by the kubelet on each node, and their traffic is carried by kube-proxy rules already programmed on those nodes. Liveness and readiness probes keep working too, because the kubelet executes them locally. What you lose in an API outage is the ability to change anything: no scheduling, no rollouts, no scaling, no kubectl.

Recover the cluster. The workload in namespace web must stay up throughout.

This exercise covers two CKA domains: Cluster Architecture, Installation and Configuration, and Troubleshooting.

The plan:

1. Confirm the diagnosis without the API, since kubectl is not coming back on its own.
2. Renew the certificates.
3. Bring the control plane back.
4. Restore your own kubectl access.
5. Verify the cluster and the workload.

Constraints:

• Do not rotate the CA. /etc/kubernetes/pki/ca.crt and ca.key stay as they are.
• Do not modify the workload in namespace web.

Pre-flight

Work on cplane-01 as root (sudo -i). Since kubectl get nodes fails, you need tools that read the certificates from disk instead of asking the API. Find out which certificates this cluster has, when each one expires, and which authority signed them.

One piece of background that explains the shape of this incident: kubeadm init issues the cluster CA with ten years of validity and every leaf certificate with one year. It issues all the leaves at the same moment, so they expire at roughly the same moment. A cluster that is upgraded regularly never notices, because kubeadm upgrade renews them as a side effect. A cluster that is left alone for a year does notice, all at once.

Step 1: Renew the certificates

Renew the kubeadm-managed certificates on cplane-01. One expired certificate caused the outage, but kubeadm issued all of them at the same time, so the others are not far behind. The check expects every kubeadm-managed certificate to be valid well past today, including the client certificates embedded in the kubeconfig files under /etc/kubernetes/.

Step 2: Bring the control plane back

A renewed file on disk is not the same as a renewed credential in use.

Nothing here crashed. The API server loaded the expired certificate quite happily and served it; it was every client that refused the handshake. So there is no crash loop to wait out, and nothing will fix itself.

Two reload behaviours differ here, and telling them apart is what this step tests:

• The API server watches its --tls-cert-file and picks up a replacement serving certificate without a restart.
• The controller-manager and the scheduler read their kubeconfig exactly once, at startup. They are still holding the credential they loaded when they started, and renewing a file on disk does not reach into a running process.

Check what port 6443 serves, then confirm the other two components are back in contact with the API server:

echo | openssl s_client -connect localhost:6443 2>/dev/null | openssl x509 -noout -enddate
kubectl -n kube-system get lease kube-controller-manager kube-scheduler \
  -o custom-columns=NAME:.metadata.name,RENEW:.spec.renewTime

The custom columns matter: the default get lease table shows only NAME, HOLDER, and AGE, and AGE is the lease object's age, not when it was last renewed. A leader-election lease that stopped being renewed is a component that cannot talk to the API server.

Step 3: Restore your own access

The control plane is back and kubectl works as root with /etc/kubernetes/admin.conf. Try it as the regular user now:

exit          # drop back from sudo -i to laborant
kubectl get nodes

It fails with error: You must be logged in to the server (the server has asked for the client to provide credentials), even though root's kubectl works fine against the same cluster. Restore your own access.

grep client-certificate-data ~/.kube/config | awk '{print $2}' | base64 -d | openssl x509 -noout -enddate
sudo grep client-certificate-data /etc/kubernetes/admin.conf | awk '{print $2}' | base64 -d | openssl x509 -noout -enddate

Step 4: Verify the cluster

Confirm the renewal did not break anything else. The nodes should be Ready, kubectl should work for both root and laborant, and the workload in web should still be serving.

Related on iximiuz Labs

Other people's work on the same terrain, worth doing alongside this one:

• Inspecting and Extracting Kubernetes Kubeconfig Data by Omkar Shelke. Reading the client certificate embedded in a kubeconfig, which is one of the certificates that expires here
• CKA Practice: Upgrade Multi-Node Kubernetes Cluster by Adam Leskis. The upgrade path that renews every certificate as a side effect, and so prevents this outage
• Kubernetes the (Very) Hard Way by Márk Sági-Kazár. Building the cluster PKI by hand, which is the best way to learn what kubeadm is renewing

References

• kubeadm certs
• Certificate management with kubeadm
• Static pods