Challenge, Medium,  on  Kubernetes, Security

Troubleshoot CrashLoopBackOff Caused by a Missing TLS Secret

Omkar Shelke
by  Omkar Shelke

Scenario

The platform team deployed api-server in the platform namespace. The application serves TLS traffic — it loads a TLS certificate and key at startup using Python's ssl module.

The deployment was applied without the TLS volume mount, so the app crashes at runtime because the certificate and key files do not exist inside the container.

The TLS certificate and key files have already been generated and are available on the dev-machine at:

/home/laborant/certs/api-server/tls.crt
/home/laborant/certs/api-server/tls.key

Task

Fix the failing deployment in two steps:

  1. Create a TLS Secret named api-server-tls in the platform namespace using the certificate files located at /home/laborant/certs/api-server/ on dev-machine.
  2. Patch the api-server Deployment to mount the secret into the server container:
    • Check the pod logs to identify the exact path the app is trying to load the TLS files from
    • Add a volume named tls-certs referencing secretName: api-server-tls
    • Mount the volume into the server container at the path you found in the logs with readOnly: true

Once both changes are applied, the Flask app will find the certificate files, load them at startup, and begin serving TLS on port 8443.

Use the following command to verify that the TLS certificates are correctly mounted and the application is serving HTTPS successfully:

kubectl run tmp --image=curlimages/curl --restart=Never --rm -it \
  -n platform -- \
  curl -vk https://api-server.platform.svc/health

Do not modify the ConfigMap or Service. Only create the secret and patch the Deployment to add the volume and volume mount.

Hint 1 — Create the TLS Secret

Use kubectl create secret tls with the certificate files on disk:

kubectl create secret tls <secret-name> \
  --cert=<path-to-tls.crt> \
  --key=<path-to-tls.key> \
  -n <namespace>
Hint 2 — Patch the Deployment

Use kubectl patch --type=strategic to append a volume and volume mount to the Deployment without replacing existing ones.

kubectl patch deployment api-server -n platform --type=strategic --patch='...'

Test Cases

How to Author Challenges on iximiuz Labs

Instead of providing a subpar online editing experience, iximiuz Labs offers a helper CLI tool called labctl, allowing you to use your favorite text editor (or a full-featured IDE) to write content from the comfort of your local machine.

Install labctl CLI

curl -sf https://labs.iximiuz.com/cli/install.sh | sh

This will download and install the latest version of the labctl CLI. You only need to do this once per workstation.

Authorize labctl

labctl auth login

This will open a browser window asking you to authorize labctl to access your account. You need to do it after a fresh install of labctl and repeat it whenever the auth session expires.

Pull challenge content

labctl content pull challenge recover-crashing-deployment-by-recreating-tls-secret-df22c665

This will create a local copy of the challenge content in a directory named recover-crashing-deployment-by-recreating-tls-secret-df22c665. You only need to do this once per challenge.

Stream your changes

labctl content push -fw challenge recover-crashing-deployment-by-recreating-tls-secret-df22c665

Run this command in a separate terminal to continuously upload your changes to the server while editing the challenge in your favorite text editor or IDE.

You can also use labctl to create, list, and delete your content. Learn more about the available commands: labctl content --help

Start challenge