Kubernetes Audit Logging

What Kubernetes Audit Logging Captures

Audit logs answer: "Who did what, to which resource, and when?" Without them, a security incident is impossible to investigate.

Level	What is recorded
`None`	Nothing
`Metadata`	User, resource, verb — no body
`Request`	Metadata + request body
`RequestResponse`	Metadata + request + response body

Step 1: Write the Audit Policy

Create the policy file the API server will load. Log secrets and RBAC changes at full detail (RequestResponse), other writes at Request, and skip noise entirely.

cat > /etc/kubernetes/audit-policy.yaml << 'EOF'
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  - level: RequestResponse
    resources:
    - group: ""
      resources: ["secrets"]

  - level: Request
    verbs: ["create", "update", "patch", "delete"]
    resources:
    - group: ""
      resources: ["pods"]

  - level: RequestResponse
    resources:
    - group: "rbac.authorization.k8s.io"
      resources: ["clusterroles", "clusterrolebindings", "roles", "rolebindings"]

  - level: None
    nonResourceURLs: ["/healthz", "/readyz", "/livez", "/metrics"]

  - level: None
    users: ["system:kube-proxy"]
    verbs: ["watch"]

  - level: Metadata
    omitStages:
    - RequestReceived
EOF

echo "Policy written."

Step 2: Enable Audit Logging in the API Server

On a kubeadm cluster, the API server runs as a static pod — its manifest lives at /etc/kubernetes/manifests/kube-apiserver.yaml. Edit that file and kubelet automatically restarts the API server.

Add three flags and two volume mounts:

python3 - << 'EOF'
import re

path = "/etc/kubernetes/manifests/kube-apiserver.yaml"
with open(path) as f:
    content = f.read()

# Add audit flags after the kube-apiserver command line
audit_flags = """    - --audit-log-path=/var/log/kubernetes/audit.log
    - --audit-log-maxage=7
    - --audit-log-maxbackup=3
    - --audit-log-maxsize=100
    - --audit-policy-file=/etc/kubernetes/audit-policy.yaml"""

# Insert after the first 'command:' block line that has 'kube-apiserver'
content = re.sub(
    r'(    - kube-apiserver\n)',
    r'\1' + audit_flags + '\n',
    content
)

# Add volumeMounts for the audit files
volume_mount = """    - mountPath: /var/log/kubernetes/audit.log
      name: audit-log
      readOnly: false
    - mountPath: /etc/kubernetes/audit-policy.yaml
      name: audit-policy
      readOnly: true"""

content = re.sub(
    r'(    volumeMounts:\n)',
    r'\1' + volume_mount + '\n',
    content
)

# Add hostPath volumes
volume_def = """  - hostPath:
      path: /var/log/kubernetes/audit.log
      type: FileOrCreate
    name: audit-log
  - hostPath:
      path: /etc/kubernetes/audit-policy.yaml
      type: File
    name: audit-policy"""

content = re.sub(
    r'(  volumes:\n)',
    r'\1' + volume_def + '\n',
    content
)

with open(path, "w") as f:
    f.write(content)

print("Manifest updated. Kubelet will restart the API server.")
EOF

Wait for the API server to come back (30–60 seconds):

echo "Waiting for API server to restart..."
sleep 10
until kubectl get nodes > /dev/null 2>&1; do
  echo "  still restarting..."
  sleep 5
done
echo "API server is back."

# Confirm the flag is active
node=$(kubectl get nodes -o jsonpath='{.items[0].metadata.name}')
kubectl get pod "kube-apiserver-${node}" -n kube-system \
  -o jsonpath='{.spec.containers[0].command}' | tr ',' '\n' | grep audit

Expected:

--audit-log-path=/var/log/kubernetes/audit.log
--audit-policy-file=/etc/kubernetes/audit-policy.yaml

Step 3: Generate Audit Events

Every API call you make is recorded. Create some high-value events:

kubectl create namespace audit-test

kubectl create secret generic db-password \
  --from-literal=password=SuperSecret123 \
  -n audit-test

kubectl get secret db-password -n audit-test -o yaml

kubectl create serviceaccount audit-sa -n audit-test

kubectl create rolebinding audit-test-binding \
  --clusterrole=view \
  --serviceaccount=audit-test:audit-sa \
  -n audit-test

Step 4: Read the Audit Log

# Confirm the file exists and has content
ls -lh /var/log/kubernetes/audit.log

# Show recent secret and RBAC events
grep '"resource":"secrets"\|"resource":"rolebindings"' \
  /var/log/kubernetes/audit.log | tail -5 | \
  python3 -c "
import sys, json
for line in sys.stdin:
    try:
        e = json.loads(line.strip())
        verb  = e.get('verb', '')
        res   = e.get('objectRef', {}).get('resource', '')
        name  = e.get('objectRef', {}).get('name', '')
        ns    = e.get('objectRef', {}).get('namespace', '')
        user  = e.get('user', {}).get('username', '')
        level = e.get('level', '')
        code  = e.get('responseStatus', {}).get('code', '')
        print(f'[{level}] {user} | {verb} {res}/{name} (ns:{ns}) -> HTTP {code}')
    except:
        pass
"

Expected output:

[RequestResponse] kubernetes-admin | create secrets/db-password (ns:audit-test) -> HTTP 201
[RequestResponse] kubernetes-admin | get secrets/db-password (ns:audit-test) -> HTTP 200
[RequestResponse] kubernetes-admin | create rolebindings/audit-test-binding (ns:audit-test) -> HTTP 201

Why This Matters

Without audit logging	With audit logging
"Someone deleted our secret" — no trace	Who deleted it, when, from which IP
RBAC misconfiguration discovered after damage	Alert fires the moment `cluster-admin` is bound
Compliance audit fails	Evidence trail satisfies SOC 2 / PCI-DSS

Cleanup

kubectl delete namespace audit-test

Course progress