Lesson  in  DevSecOps in Practice: Container & Kubernetes Security

HashiCorp Vault + External Secrets Operator

Deploy Vault, store database credentials and API keys in it, then install the External Secrets Operator to sync them into Kubernetes Secrets automatically. Update the secret in Vault and watch Kubernetes rotate it — no manual intervention required.

Deploy Vault & Store Secrets

The Problem Vault Solves

Kubernetes Secrets live in etcd as base64 — that does not change with Vault. The External Secrets Operator still syncs secrets into standard K8s Secrets, so they do land in etcd.

What Vault changes is where secrets are managed and who controls them:

Plain K8s SecretsVault + ESO
Source of truthetcd (inside cluster)Vault (external, dedicated secrets store)
RotationManual — edit the Secret, restart podsVault rotates; ESO re-syncs automatically
Audit logKubernetes audit log (if enabled)Every read/write logged in Vault
Access controlKubernetes RBACVault policies, auth backends (LDAP, AppRole, AWS IAM)
Secret sprawlOne copy per namespace/clusterSingle source across clusters and services

The security gain is not avoiding etcd — it is having a dedicated, audited, centrally managed secrets store that your entire organisation shares, rather than secrets scattered across dozens of clusters and namespaces.

Deploy HashiCorp Vault

helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update

helm install vault hashicorp/vault \
  --namespace vault \
  --create-namespace \
  --set "server.dev.enabled=true" \
  --set "server.dev.devRootToken=root" \
  --set "injector.enabled=false"

kubectl wait --for=condition=Ready pod/vault-0 -n vault --timeout=120s
echo "Vault is running."

Dev mode — single-node, no persistence, no auth complexity. For demos only.

Verify Vault status

kubectl exec -n vault vault-0 -- vault status

Expected: Sealed: false

Store Secrets in Vault

kubectl exec -n vault vault-0 -- vault secrets enable -path=secret kv-v2 2>/dev/null || \
  echo "KV engine already enabled"

kubectl exec -n vault vault-0 -- \
  vault kv put secret/myapp/database \
    username="appuser" \
    password="Vault\$ecure#2025" \
    host="postgres.internal.dangote.com" \
    port="5432"

kubectl exec -n vault vault-0 -- \
  vault kv put secret/myapp/api-keys \
    payment-gateway="pg_live_abc123xyz789" \
    sms-provider="sms_key_def456uvw"

echo "Secrets stored."

Read back to confirm

kubectl exec -n vault vault-0 -- vault kv get secret/myapp/database

Store the Vault Token for ESO

ESO needs a credential to authenticate to Vault. We store the dev root token as a K8s Secret.

kubectl create namespace app-demo

kubectl create secret generic vault-token \
  --from-literal=token=root \
  -n app-demo

echo "Vault token stored."

External Secrets Operator & Auto-rotation

Install the External Secrets Operator

ESO watches for ExternalSecret resources and syncs secrets from Vault (or Azure Key Vault, AWS, GCP) into Kubernetes Secrets.

kubectl create namespace app-demo

helm repo add external-secrets https://charts.external-secrets.io
helm repo update

helm install external-secrets external-secrets/external-secrets \
  --namespace external-secrets \
  --create-namespace \
  --set installCRDs=true \
  --wait --timeout 3m

Wait for the CRDs to be fully established — --wait checks pods but not CRD registration:

kubectl wait --for condition=established --timeout=120s \
  crd/secretstores.external-secrets.io \
  crd/externalsecrets.external-secrets.io \
  crd/clustersecretstores.external-secrets.io

kubectl get pods -n external-secrets

Connect ESO to Vault

Verify Vault is Reachable

kubectl exec -n vault vault-0 -- wget -qO- http://localhost:8200/v1/sys/health

Expected: JSON with "initialized":true,"sealed":false.

Create a SecretStore

If you are retrying this step, delete any previous SecretStore first:

kubectl delete secretstore vault-backend -n app-demo --ignore-not-found

cat > secret-store.yaml << 'EOF'
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: vault-backend
  namespace: app-demo
spec:
  provider:
    vault:
      server: "http://vault.vault.svc.cluster.local:8200"
      path: "secret"
      version: "v2"
      auth:
        tokenSecretRef:
          name: vault-token
          namespace: app-demo
          key: token
EOF

kubectl apply -f secret-store.yaml
sleep 8
kubectl get secretstore vault-backend -n app-demo

Expected: STATUS=Valid, READY=True

If you see InvalidProviderConfig, check details with:

kubectl describe secretstore vault-backend -n app-demo | grep -A10 "Status:"

Create an ExternalSecret

cat > external-secret.yaml << 'EOF'
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: database-credentials
  namespace: app-demo
spec:
  refreshInterval: 1m
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: db-credentials
    creationPolicy: Owner
  data:
  - secretKey: DB_USERNAME
    remoteRef:
      key: myapp/database
      property: username
  - secretKey: DB_PASSWORD
    remoteRef:
      key: myapp/database
      property: password
  - secretKey: DB_HOST
    remoteRef:
      key: myapp/database
      property: host
EOF

kubectl apply -f external-secret.yaml
sleep 10
kubectl get externalsecret database-credentials -n app-demo

Expected: STATUS=SecretSynced, READY=True

kubectl get secret db-credentials -n app-demo -o jsonpath='{.data.DB_HOST}' | base64 -d
echo ""

Expected: the hostname stored in Vault.

Simulate Secret Rotation

Update the password in Vault. ESO picks it up within 1 minute — no pod restart needed.

kubectl exec -n vault vault-0 -- \
  vault kv put secret/myapp/database \
    username="appuser" \
    password="NewRotated\$ecure#2025" \
    host="postgres.internal.example.com" \
    port="5432"

echo "Waiting for ESO to sync (1 minute)..."
sleep 65

kubectl get secret db-credentials -n app-demo \
  -o jsonpath='{.data.DB_PASSWORD}' | base64 -d
echo ""

Expected: NewRotated$ecure#2025 — the K8s Secret rotated automatically without touching a manifest or restarting any pod.

Architecture

Vault (source of truth)
    │
    │  ESO syncs every 1m
    ▼
Kubernetes Secret (auto-created, auto-rotated)
    │
    ▼
Application Pod (uses normal K8s Secret API)

The application doesn't know Vault exists. It reads a normal K8s Secret. ESO handles everything.

Cleanup

helm uninstall vault -n vault 2>/dev/null || true
helm uninstall external-secrets -n external-secrets 2>/dev/null || true
kubectl delete namespace app-demo external-secrets vault --ignore-not-found
Previous lesson
Kubernetes Secrets: Safe Consumption
Next lesson
Pod Hardening & Pod Security Admission
Start lesson