Roll Out Pod Security Standards Across Three Namespaces

Scenario

The platform team is rolling out Pod Security Admission, the built-in admission controller that applies the Kubernetes Pod Security Standards per namespace. Three namespaces are in scope this week:

Namespace	Workload	What it is
`monitoring`	DaemonSet `node-agent`	Node metrics agent. It reads host processes, so it uses the host network and PID namespaces and mounts the host's `/proc`.
`storefront`	Deployment `storefront-web`	Public website, 2 replicas. Its team has not hardened the image yet.
`payments`	Deployment `payments-api`	Card payments API, 2 replicas. It handles card data and has to meet the strictest standard.

The monitoring namespace has a Pod Security label, and the node-agent DaemonSet cannot create any Pods. It is down on every node. storefront and payments have no Pod Security labels yet.

Task

Fix the monitoring namespace by setting the appropriate Pod Security level, so that the node-agent DaemonSet runs a Pod on every node. The agent needs hostNetwork, hostPID and a hostPath volume, so do not change the DaemonSet.
Configure the storefront namespace to:
- Block Pods that break the baseline level.
- Allow Pods that break the restricted level, but show a warning and record them in the audit log.
- The storefront-web Deployment must keep running.
Make the payments namespace enforce the restricted level. Then update the payments-api Deployment so that its Pods follow that level and it shows 2/2 Ready.

Note

Work from cplane-01. The API server on that machine writes its audit log to /var/log/kubernetes/audit.log, which needs sudo to read.

Important

Set every policy with the Pod Security Admission labels on the namespace. For the payments-api Deployment, change its Pod spec, not its image, and do not add exemptions. Leave the node-agent DaemonSet and the storefront-web Deployment as they are.

Hint 1 | Finding out why a controller creates no Pods

When an admission controller rejects a Pod, the controller that tried to create it records the reason as an event:

kubectl describe daemonset <name> -n <namespace>
kubectl get events -n <namespace> --field-selector reason=FailedCreate
kubectl get namespace <namespace> --show-labels

Hint 2 | The shape of a Pod Security label

Each namespace label sets one mode to one level. A namespace can carry one label per mode:

pod-security.kubernetes.io/<MODE>=<LEVEL>

kubectl label namespace <namespace> <label>=<value> --overwrite

A server-side dry run shows which running Pods a new enforce level would reject, without changing anything:

kubectl label --dry-run=server --overwrite namespace <namespace> <label>=<value>

See: Pod Security Standards, Pod Security Admission

Hint 3 | When a running Deployment is not compliant

A new enforce level never touches Pods that are already running, so a Deployment can look healthy while every new Pod is rejected. Force a rollout and see what the ReplicaSet reports:

kubectl rollout restart deployment/<name> -n <namespace>
kubectl get events -n <namespace> --field-selector reason=FailedCreate

The rejection message names each field the Pod spec needs. It says pod or container for a field that exists at both levels, and container for one that exists only on the container:

spec:
  securityContext:        # Pod level, applies to every container
    <field>: <value>
  containers:
  - name: <name>
    securityContext:      # container level, for this container only
      <field>: <value>

Hint 4 | Seeing what audit mode recorded

Audit mode does not print anything. It adds an annotation to the matching entry in the API server audit log:

sudo grep -c <annotation> /var/log/kubernetes/audit.log

Tasks

Roll Out Pod Security Standards Across Three Namespaces

Scenario

Task

⚒ Test Cases