Lesson  in  DevSecOps in Practice: Container & Kubernetes Security

Policy-as-Code with Kyverno

RBAC controls who can do things. Kyverno controls what can be done — at admission time, before any container ever starts. Write policies that block the latest image tag (unpinned versions are a supply-chain risk), require resource limits on every container, and mandate non-root execution. Then prove that non-compliant workloads are rejected by the API server before they reach the scheduler.

Installing Kyverno & Writing Admission Policies

What Kyverno Does

RBAC controls who can create resources. Kyverno controls what those resources can look like — before they reach the scheduler, before any container starts.

Kyverno runs as a Kubernetes admission webhook: every kubectl apply passes through it. If the manifest violates a policy, the API server returns an error and the resource is never created.

kubectl apply -f pod.yaml
        │
        ▼
  API Server
        │
        ▼
  Kyverno Webhook ──── Policy: no latest tag
        │                        Policy: require resource limits
        │                        Policy: require non-root
        ▼
  Reject (HTTP 403)  OR  Accept → Scheduler

Kyverno also supports mutating policies: automatically add labels, inject sidecars, or set default values — making the secure path the effortless path.

Install Kyverno

helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update

helm upgrade --install kyverno kyverno/kyverno \
  --namespace kyverno \
  --create-namespace \
  --wait \
  --timeout 5m

# Verify
kubectl get pods -n kyverno

Policy 1: Disallow the latest Tag

latest is a moving target. An image tagged latest today may be different from latest tomorrow — making deployments non-reproducible and vulnerable to supply-chain substitution attacks.

cat > policy-no-latest.yaml << 'EOF'
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-latest-tag
  annotations:
    policies.kyverno.io/title: Disallow Latest Tag
    policies.kyverno.io/description: >
      Require image tags and disallow 'latest' as an image tag to ensure
      the image is pinned to a specific version.
spec:
  validationFailureAction: Enforce
  background: true
  rules:
  - name: require-image-tag
    match:
      any:
      - resources:
          kinds: [Pod]
    validate:
      message: "An image tag is required and 'latest' is not allowed."
      pattern:
        spec:
          containers:
          - image: "*:*"
  - name: validate-image-tag
    match:
      any:
      - resources:
          kinds: [Pod]
    validate:
      message: "Using a mutable image tag such as 'latest' is not allowed."
      deny:
        conditions:
          any:
          - key: "{{request.object.spec.containers[].image}}"
            operator: AnyIn
            value: ["*:latest"]
EOF

kubectl apply -f policy-no-latest.yaml

Policy 2: Require Resource Limits

cat > policy-require-limits.yaml << 'EOF'
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-resource-limits
  annotations:
    policies.kyverno.io/title: Require Resource Limits
    policies.kyverno.io/description: >
      Resource limits must be set to prevent a single container from consuming
      all available cluster resources (denial-of-service).
spec:
  validationFailureAction: Enforce
  background: true
  rules:
  - name: validate-limits
    match:
      any:
      - resources:
          kinds: [Pod]
    validate:
      message: "Resource limits (cpu and memory) are required for all containers."
      pattern:
        spec:
          containers:
          - resources:
              limits:
                memory: "?*"
                cpu: "?*"
EOF

kubectl apply -f policy-require-limits.yaml

Policy 3: Require Non-Root Execution

cat > policy-require-nonroot.yaml << 'EOF'
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-non-root
  annotations:
    policies.kyverno.io/title: Require Non-Root User
    policies.kyverno.io/description: >
      Containers must not run as root (UID 0). Root inside a container is
      root on the host if container escape occurs.
spec:
  validationFailureAction: Enforce
  background: true
  rules:
  - name: check-runasnonroot
    match:
      any:
      - resources:
          kinds: [Pod]
          namespaces: ["default", "production", "staging"]
    validate:
      message: "Containers must not run as root. Set runAsNonRoot: true or runAsUser > 0."
      anyPattern:
      - spec:
          securityContext:
            runAsNonRoot: true
      - spec:
          containers:
          - securityContext:
              runAsUser: ">0"
EOF

kubectl apply -f policy-require-nonroot.yaml

Testing Policies — Blocking Non-Compliant Workloads

The Moment of Truth

Policies are only useful if they actually block non-compliant workloads. Let's test each one:

Test 1: latest Tag Blocked

kubectl apply -f - << 'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: test-latest
  namespace: default
spec:
  containers:
  - name: app
    image: nginx:latest
    resources:
      limits: {cpu: "100m", memory: "64Mi"}
      requests: {cpu: "50m", memory: "32Mi"}
EOF

Expected:

Error from server: admission webhook "validate.kyverno.svc-fail" denied the request:
resource Pod/default/test-latest was blocked due to the following policies:
disallow-latest-tag:
  validate-image-tag: Using a mutable image tag such as 'latest' is not allowed.

Test 2: Missing Resource Limits Blocked

kubectl apply -f - << 'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: test-no-limits
  namespace: default
spec:
  containers:
  - name: app
    image: public.ecr.aws/nginx/nginx:1.25-alpine
EOF

Expected:

...was blocked due to the following policies:
require-resource-limits:
  validate-limits: Resource limits (cpu and memory) are required for all containers.

Test 3: Root User Blocked

kubectl apply -f - << 'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: test-root
  namespace: default
spec:
  containers:
  - name: app
    image: public.ecr.aws/nginx/nginx:1.25-alpine
    securityContext:
      runAsUser: 0
    resources:
      limits: {cpu: "100m", memory: "64Mi"}
      requests: {cpu: "50m", memory: "32Mi"}
EOF

Expected: Rejected due to require-non-root policy.

Deploy a Compliant Pod

Now deploy something that passes all three policies:

kubectl apply -f - << 'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: compliant-app
  namespace: default
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
  containers:
  - name: app
    image: public.ecr.aws/nginx/nginx:1.25-alpine
    resources:
      requests: {cpu: "100m", memory: "64Mi"}
      limits: {cpu: "200m", memory: "128Mi"}
EOF

kubectl get pod compliant-app

Expected: Pod created successfully and reaches Running.

Audit Mode vs Enforce Mode

Our policies use validationFailureAction: Enforce — they block non-compliant resources. Kyverno also supports Audit mode:

validationFailureAction: Audit

In Audit mode, non-compliant resources are allowed but flagged in a PolicyReport. This is useful when rolling out policies to an existing cluster — you can see what would be blocked before actually blocking it:

kubectl get policyreport -A
kubectl get clusterpolicyreport

The Kyverno + RBAC + Pod Security Relationship

Who can do what?        → RBAC (companion course Module 4)
What workloads allowed? → Kyverno ClusterPolicies (this lab)
What can a pod do?      → Pod Security Standards (companion course Module 2)
What can a pod access?  → NetworkPolicy (companion course Module 3)

Together these four layers form defence in depth: an attacker must bypass all four to achieve persistence.

Cleanup

kubectl delete pod compliant-app --ignore-not-found
kubectl delete clusterpolicy disallow-latest-tag require-resource-limits require-non-root
Previous lesson
GitOps with Argo CD — Every Change Auditable
Next lesson
Helm — Packaging Security Into Your App Template
Start lesson