Lesson  in  DevSecOps in Practice: Container & Kubernetes Security

Rolling Updates — Zero-Downtime CVE Patching

A vulnerability is discovered in your running container image. In a traditional server environment, patching means downtime. In Kubernetes, a rolling update replaces every pod one by one while traffic keeps flowing. This lab scans a running deployment for CVEs, patches it to a clean image, and proves no requests were dropped during the upgrade.

Deploying a Vulnerable Image & Scanning It

The Real DevSecOps Challenge: Running Systems

The hardest part of vulnerability management is not scanning images before deployment — it is finding and patching CVEs in already-running workloads without taking the service down.

A traditional server environment forces a choice: live with the vulnerability, or schedule downtime. Kubernetes removes that choice entirely. A rolling update replaces pods one at a time while traffic continues to flow.

Create the Demo Namespace

kubectl create namespace patching-demo

Deploy the Vulnerable Image

We will deploy public.ecr.aws/nginx/nginx:1.21, a version with known CVEs:

cat > webapp.yaml << 'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: webapp
  namespace: patching-demo
  annotations:
    deployment.kubernetes.io/revision: "1"
spec:
  replicas: 3
  selector:
    matchLabels:
      app: webapp
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  template:
    metadata:
      labels:
        app: webapp
    spec:
      containers:
      - name: nginx
        image: public.ecr.aws/nginx/nginx:1.21
        resources:
          requests: {cpu: "100m", memory: "64Mi"}
          limits: {cpu: "200m", memory: "128Mi"}
        ports:
        - containerPort: 80
EOF

kubectl apply -f webapp.yaml
kubectl rollout status deployment/webapp -n patching-demo --timeout=90s

# Keep a load generator running in the background
kubectl run load-gen \
  --image=public.ecr.aws/docker/library/busybox:1.36 \
  --restart=Never \
  -n patching-demo \
  -- sh -c "while true; do wget -qO- http://webapp.patching-demo.svc.cluster.local 2>/dev/null; sleep 0.5; done" &

Expose the Service

cat > webapp-svc.yaml << 'EOF'
apiVersion: v1
kind: Service
metadata:
  name: webapp
  namespace: patching-demo
spec:
  selector:
    app: webapp
  ports:
  - port: 80
    targetPort: 80
EOF

kubectl apply -f webapp-svc.yaml

Scan the Running Image with Trivy

Trivy can scan the same image currently running in your cluster:

# Find the exact image being used
IMAGE=$(kubectl get deployment webapp -n patching-demo \
  -o jsonpath='{.spec.template.spec.containers[0].image}')
echo "Scanning: $IMAGE"

# Scan for HIGH and CRITICAL CVEs
trivy image --severity HIGH,CRITICAL --no-progress "$IMAGE"

The output will show CVEs categorised by package and severity. Note the CRITICAL column — those are the ones that need immediate patching.

# Count total HIGH+CRITICAL
trivy image --severity HIGH,CRITICAL --no-progress --quiet "$IMAGE" 2>/dev/null | \
  grep -E "^Total:" || echo "Scan complete — see output above"

Rolling Update — Patch Live, Zero Downtime

How a Rolling Update Works

The RollingUpdate strategy (set in the previous unit) updates pods incrementally:

Before:  [Pod v1] [Pod v1] [Pod v1]
Step 1:  [Pod v1] [Pod v1] [Pod v2]  ← 1 new pod started (maxSurge: 1)
Step 2:  [Pod v1] [Pod v2] [Pod v2]  ← 1 old pod terminated (maxUnavailable: 1)
Step 3:  [Pod v2] [Pod v2] [Pod v2]  ← complete

At no point are all pods down simultaneously. Traffic flows to healthy pods throughout.

Patch the Image

# Option 1: kubectl set image (fast, for emergencies)
kubectl set image deployment/webapp \
  nginx=public.ecr.aws/nginx/nginx:1.25-alpine \
  -n patching-demo

# Option 2: Update the YAML and apply (preferred for GitOps)
# sed -i 's|public.ecr.aws/nginx/nginx:1.21|public.ecr.aws/nginx/nginx:1.25-alpine|' webapp.yaml
# kubectl apply -f webapp.yaml

Watch the Rollout in Real Time

Open a second terminal and watch pods being replaced:

kubectl get pods -n patching-demo -w

In the first terminal, watch the rollout status:

kubectl rollout status deployment/webapp -n patching-demo

Expected output:

Waiting for deployment "webapp" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "webapp" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "webapp" rollout to finish: 1 old replicas are pending termination...
deployment "webapp" successfully rolled out

Verify Fewer CVEs in the New Image

trivy image --severity HIGH,CRITICAL --no-progress --quiet public.ecr.aws/nginx/nginx:1.25-alpine

The count will be dramatically lower than public.ecr.aws/nginx/nginx:1.21. Alpine's minimal base eliminates most of the OS-level CVE surface that the Debian-based image carries. The exact number depends on the vulnerability database at the time you run the scan — CVEs are discovered continuously, so even well-maintained images may show some findings on any given day. The goal is the comparison, not absolute zero.

Scan the Running Image Again

NEW_IMAGE=$(kubectl get deployment webapp -n patching-demo \
  -o jsonpath='{.spec.template.spec.containers[0].image}')
echo "Now running: $NEW_IMAGE"
trivy image --severity HIGH,CRITICAL --no-progress "$NEW_IMAGE"

The DevSecOps Workflow

Trivy finds CVE in running image
        │
        ▼
Update image tag in git (PR with Trivy scan results)
        │
        ▼
CI pipeline scans new image: trivy --exit-code 1
        │
        ▼
kubectl apply (or Argo CD auto-syncs — next module)
        │
        ▼
Rolling update: zero downtime, full audit trail
        │
        ▼
Post-deploy scan confirms 0 HIGH/CRITICAL CVEs

This is exactly what the companion course builds on: automated scanning gates, GitOps-driven rollouts, and runtime detection when a CVE is exploited before you patch it.

Cleanup

kubectl delete namespace patching-demo
Previous lesson
Resource Limits & Denial-of-Service Prevention
Next lesson
Health Probes & Self-Healing
Start lesson