Drill — the Secret that was rotated but never reloaded

Drill — this is a synthetic composite of a failure mode reported across many production clusters, not a specific company's incident. The full pattern write-up: Pattern: Secret rotated, never reloaded.

The situation

Security rotated the database password an hour ago. The Secret is correct:

kubectl -n kubelings get secret db-creds -o jsonpath='{.data.password}' | base64 -d
# s3cure-NEW-9917

The billing pod disagrees:

kubectl -n kubelings logs billing --tail=1
# connecting to db with password=hunter2-2024Q4

Same cluster, same Secret, two different truths. The pod spec explains it:

kubectl -n kubelings get pod billing -o jsonpath='{.spec.containers[0].env}'

DB_PASSWORD comes from secretKeyRef — an env var. Env vars are resolved once, when the container starts, copied into the process environment, and never touched again. The Secret can rotate hourly; the process keeps the value it was born with. (The old password gets revoked next, and then billing goes down — at the revocation, not the rotation, which makes the postmortem extra confusing.)

Mounted Secret volumes behave differently: the kubelet re-syncs them (~1 min), and the file content updates in place, no restart.

                                      ┌─────────────┐
                    ┌─────mounted────▶│file updates │
 ┌───────────────┐──┘                 │             │
 │Secret rotated │                    └─────────────┘
 │               │──┐                 ┌───────────┐  
 └───────────────┘  └─────envFrom────▶│env frozen │  
                                      │           │  
                                      └───────────┘  

Your task

Recreate billing consuming db-creds as a volume mount instead of env:

containers:
  - name: billing
    volumeMounts:
      - {name: creds, mountPath: /etc/db-creds, readOnly: true}
volumes:
  - name: creds
    secret: {secretName: db-creds}

Have the app read the file (cat /etc/db-creds/password) instead of $DB_PASSWORD. Pod env is immutable — this is a delete-and-recreate (kubectl get pod billing -o yaml → edit → kubectl replace --force), or write the pod fresh.

kubectl -n kubelings delete pod billing
kubectl apply -n kubelings -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: billing
  labels: {app: billing}
spec:
  containers:
    - name: billing
      image: busybox:1.36
      command: ["sh", "-c", "while true; do echo \"connecting to db with password=$(cat /etc/db-creds/password)\"; sleep 15; done"]
      volumeMounts:
        - {name: creds, mountPath: /etc/db-creds, readOnly: true}
  volumes:
    - name: creds
      secret: {secretName: db-creds}
EOF

kubectl -n kubelings exec billing -- cat /etc/db-creds/password
# s3cure-NEW-9917
kubectl -n kubelings logs billing --tail=1