Lesson  in  DevSecOps in Practice: Container & Kubernetes Security

Health Probes & Self-Healing

Without health probes, Kubernetes routes traffic to pods that are starting up, overloaded, or broken — silently dropping requests. With liveness and readiness probes configured, Kubernetes self-heals: it restarts unhealthy pods and stops routing traffic to pods that aren't ready. Availability is part of the CIA triad — this lab shows how Kubernetes enforces it automatically.

Without Probes — The Silent Failure

Availability Is Part of the CIA Triad

CIA stands for Confidentiality, Integrity, Availability. Most DevSecOps content focuses on C and I. Availability is often forgotten — until an incident makes it obvious.

Without health probes, Kubernetes has no way to know whether your application is actually working. It only knows whether the container process is running. A container can be running while:

  • Still initialising (not yet ready to serve traffic)
  • In a crash loop (dying and restarting repeatedly)
  • Deadlocked (process running, but not responding)
  • Out of memory (responding slowly, dropping requests)

In all four cases, Kubernetes will happily route traffic to the broken pod.

The Three Probes

ProbePurposeWhat happens on failure
Startup"Is the app still starting?"Container is not killed during slow startup
Liveness"Is the app still healthy?"Container is restarted
Readiness"Is the app ready for traffic?"Pod removed from Service endpoints

Create the Demo Namespace

kubectl create namespace probes-demo

Deploy Without Probes

cat > no-probe-app.yaml << 'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: no-probe-app
  namespace: probes-demo
spec:
  replicas: 2
  selector:
    matchLabels:
      app: no-probe-app
  template:
    metadata:
      labels:
        app: no-probe-app
    spec:
      containers:
      - name: app
        image: public.ecr.aws/nginx/nginx:1.25-alpine
        resources:
          requests: {cpu: "100m", memory: "64Mi"}
          limits: {cpu: "200m", memory: "128Mi"}
        ports:
        - containerPort: 80
EOF

kubectl apply -f no-probe-app.yaml
kubectl rollout status deployment/no-probe-app -n probes-demo --timeout=60s

Demonstrate the Problem

POD=$(kubectl get pods -n probes-demo -l app=no-probe-app \
  -o jsonpath='{.items[0].metadata.name}')

# Kill the nginx process inside the container
kubectl exec -n probes-demo "$POD" -- nginx -s stop

# Watch: Kubernetes sees the process die and restarts the container
kubectl get pod "$POD" -n probes-demo -w

Expected: RESTARTS column increments. Without a liveness probe, Kubernetes only restarts the container when the process exits — not when it is hanging or deadlocked.

The broader problem is readiness: for the duration between the process dying and the restart completing (several seconds), traffic is still routed to this pod, causing errors for users.

Adding Probes — Self-Healing with Precision

The Probed Deployment

cat > probe-app.yaml << 'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: probe-app
  namespace: probes-demo
spec:
  replicas: 2
  selector:
    matchLabels:
      app: probe-app
  template:
    metadata:
      labels:
        app: probe-app
    spec:
      containers:
      - name: app
        image: public.ecr.aws/nginx/nginx:1.25-alpine
        resources:
          requests: {cpu: "100m", memory: "64Mi"}
          limits: {cpu: "200m", memory: "128Mi"}
        ports:
        - containerPort: 80
        startupProbe:
          httpGet:
            path: /
            port: 80
          failureThreshold: 30
          periodSeconds: 3
        livenessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 5
          periodSeconds: 10
          failureThreshold: 3
          timeoutSeconds: 5
        readinessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 3
          periodSeconds: 5
          failureThreshold: 2
          timeoutSeconds: 3
EOF

kubectl apply -f probe-app.yaml
kubectl rollout status deployment/probe-app -n probes-demo --timeout=90s

Understanding Each Probe Setting

startupProbe:
  failureThreshold: 30   # Give the app 30 × 3s = 90s to start
  periodSeconds: 3       # Check every 3 seconds

livenessProbe:
  initialDelaySeconds: 5  # Wait 5s before first check (startup complete)
  periodSeconds: 10       # Check every 10s
  failureThreshold: 3     # Restart after 3 consecutive failures = 30s

readinessProbe:
  periodSeconds: 5        # Check every 5s
  failureThreshold: 2     # Remove from Service after 2 failures = 10s

The startup probe prevents liveness from killing a slow-starting container (a common footgun: liveness kills the app before it finishes initialising, causing a restart loop).

Watch Self-Healing in Action

cat > probe-svc.yaml << 'EOF'
apiVersion: v1
kind: Service
metadata:
  name: probe-app
  namespace: probes-demo
spec:
  selector:
    app: probe-app
  ports:
  - port: 80
    targetPort: 80
EOF

kubectl apply -f probe-svc.yaml

POD=$(kubectl get pods -n probes-demo -l app=probe-app \
  -o jsonpath='{.items[0].metadata.name}')

# Watch pods in background
kubectl get pods -n probes-demo -w &

# Watch endpoints in background
kubectl get endpoints probe-app -n probes-demo -w &

# Kill the nginx process to trigger probes
kubectl exec -n probes-demo "$POD" -- nginx -s stop

What you will see:

  1. Pod's READY drops from 1/1 to 0/1 (readiness probe fails)
  2. Pod's IP disappears from the Endpoints list — traffic stops going to it
  3. After 3 liveness failures, pod restarts
  4. Pod passes readiness checks and re-joins the Endpoints
  5. Traffic resumes to all pods

Compare this to the no-probe-app: without readiness, the Service would have continued routing traffic to the broken pod throughout the restart cycle.

Security Implications

ProbeSecurity Benefit
ReadinessPrevents traffic to pods under load (a pod being exploited may respond slowly — readiness gates it out)
LivenessLimits attacker dwell time: a crashed process gets restarted quickly, limiting how long an attacker maintains a foothold
StartupPrevents restart loops that create a window where a half-initialised app with partial security controls is reachable

Probe Best Practices

DO: Use a dedicated /healthz endpoint that checks internal state
DO: Set timeoutSeconds to prevent probe pileup under load
DO: Use startupProbe for slow-starting apps
DO NOT: Use liveness on anything that fails under load (causes cascading restarts)
DO NOT: Make liveness probe call external dependencies (DB down ≠ restart the app)

Cleanup

kubectl delete namespace probes-demo
Previous lesson
Rolling Updates — Zero-Downtime CVE Patching
Next lesson
ConfigMaps & Secrets — The Right and Wrong Way
Start lesson