Observability — Detecting Security Anomalies with Prometheus

Why Observability Is a Security Control

The NIST Cybersecurity Framework defines five functions: Identify, Protect, Detect, Respond, Recover. Most Kubernetes security controls (RBAC, Pod Security, Network Policy) address Protect. Prometheus addresses Detect.

Without metrics, you cannot answer:

• Is a pod restarting repeatedly? (crash loop = active exploit or memory leak)
• Is CPU spiking on a quiet service? (cryptominer or runaway process)
• Is a container hitting its memory limit? (DoS attack or memory exfiltration)
• Are there unusual API server requests? (privilege escalation attempt)

Prometheus scrapes these metrics every 15 seconds. Alertmanager fires when thresholds are crossed. Grafana makes it human-readable.

Architecture Overview

┌─────────────────────────────────────────────────────────┐
│                      Kubernetes Cluster                 │
│                                                         │
│  kubelet                                                │
│  (exposes /metrics on each node)                        │
│       │                                                 │
│       ▼                                                 │
│  kube-state-metrics                                     │
│  (converts K8s API objects to metrics)                  │
│       │                                                 │
│       ▼                                                 │
│  Prometheus ──────────────────→ Alertmanager            │
│  (scrapes & stores time-series)   (fires alerts)        │
│       │                                                 │
│       ▼                                                 │
│  Grafana                                                │
│  (dashboards & visualisation)                           │
└─────────────────────────────────────────────────────────┘

Install with the kube-prometheus-stack Helm Chart

The kube-prometheus-stack chart deploys the full stack (Prometheus Operator, Prometheus, Grafana, Alertmanager, kube-state-metrics, node-exporter) with production-ready defaults:

helm repo add prometheus-community \
  https://prometheus-community.github.io/helm-charts
helm repo update

kubectl create namespace monitoring

helm install kube-prometheus-stack \
  prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --set prometheus.prometheusSpec.retention=24h \
  --set grafana.adminPassword=DevSecOps2024 \
  --set alertmanager.enabled=true \
  --wait \
  --timeout 5m

This takes 2-3 minutes. The --wait flag blocks until all pods are ready.

# Verify everything is running
kubectl get pods -n monitoring

Expected: All pods in Running or Completed state.

Query Metrics via CLI

# Port-forward Prometheus to localhost
kubectl port-forward -n monitoring \
  svc/kube-prometheus-stack-prometheus 9090:9090 &

sleep 3

# Query: is everything up?
curl -s 'http://localhost:9090/api/v1/query?query=up' | \
  jq -r '.data.result[] | "\(.metric.job): \(if .value[1] == "1" then "UP" else "DOWN" end)"'

Expected: A list of scraped jobs, all showing UP.

Security-Relevant Metrics You Now Have

# Pod restart count (crash loops, OOMKills from exploits)
curl -s 'http://localhost:9090/api/v1/query?query=kube_pod_container_status_restarts_total' | \
  jq -r '.data.result[:5][] | "\(.metric.pod): \(.value[1]) restarts"'

# Containers NOT running (might be crash-looping after attack)
curl -s 'http://localhost:9090/api/v1/query?query=kube_pod_container_status_running==0' | \
  jq '.data.result | length | "Non-running containers: \(.)"'

# Memory usage vs limits (near-limit = DoS or memory leak)
curl -s 'http://localhost:9090/api/v1/query?query=container_memory_usage_bytes/container_spec_memory_limit_bytes>0.8' | \
  jq '.data.result | length | "Containers >80% memory limit: \(.)"'

The Security Signal You're Looking For

A pod that restarts repeatedly is a red flag: it could be an active exploit crashing the process, an OOMKill from a memory-based DoS attack, or a container escape attempt that terminates the process. Prometheus captures these signals automatically — no extra agents, no configuration beyond what you already deployed.

Simulate a Crash Loop

Deploy a pod that immediately exits:

kubectl run crasher \
  --image=public.ecr.aws/docker/library/busybox:1.36 \
  --restart=Always \
  -n monitoring \
  -- sh -c "echo 'process failed'; exit 1"

Watch it restart:

kubectl get pod crasher -n monitoring -w

You will see the RESTARTS column increment every 10–20 seconds. Hit Ctrl+C after 2–3 restarts.

Detect It in Prometheus

Prometheus has already scraped the restart signal. Port-forward Prometheus and query it:

kubectl port-forward -n monitoring \
  svc/kube-prometheus-stack-prometheus 9090:9090 &>/dev/null &
sleep 3

curl -sG "http://localhost:9090/api/v1/query" \
  --data-urlencode 'query=kube_pod_container_status_restarts_total{pod="crasher"}' | \
  jq -r '.data.result[] | "Restarts: \(.value[1])"'

The number matches what kubectl get pod showed. This is exactly the signal that fires a PodCrashLooping alert in a production cluster — and you can see it without touching a dashboard.

Simulate an OOMKill

Memory exhaustion is a common DoS vector — flood a container's memory until the kernel kills it. Set an absurdly low memory limit to trigger the same kernel signal:

cat > oom-pod.yaml << 'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: oom-demo
  namespace: monitoring
spec:
  containers:
  - name: app
    image: public.ecr.aws/docker/library/busybox:1.36
    resources:
      limits:
        memory: "10Mi"
    command: ["sh", "-c", "dd if=/dev/zero of=/dev/shm/fill bs=1M count=50"]
EOF
kubectl apply -f oom-pod.yaml

# Confirm the kernel killed it
kubectl describe pod oom-demo -n monitoring | grep -A3 "Last State"

Expected: Reason: OOMKilled or Reason: StartError — both mean the kernel enforced the memory limit and terminated the container.

What Every Security Anomaly Looks Like in Metrics

Crash loop  →  kube_pod_container_status_restarts_total spikes  (Prometheus)
OOMKill     →  kubectl describe pod — Last State: Terminated, Reason: OOMKilled
CPU spike   →  rate(container_cpu_usage_seconds_total[5m]) anomaly  (Prometheus)

Every attack that affects a running process produces a metric. Prometheus collects them. AlertManager fires when thresholds are crossed — you get paged before the service goes down.

Cleanup

kubectl delete pod crasher oom-demo -n monitoring