Roll Out Pod Security Standards Across Three Namespaces
Scenario
The platform team is rolling out Pod Security Admission, the built-in admission controller that applies the Kubernetes Pod Security Standards per namespace. Three namespaces are in scope this week:
| Namespace | Workload | What it is |
|---|---|---|
monitoring | DaemonSet node-agent | Node metrics agent. It reads host processes, so it uses the host network and PID namespaces and mounts the host's /proc. |
storefront | Deployment storefront-web | Public website, 2 replicas. Its team has not hardened the image yet. |
payments | Deployment payments-api | Card payments API, 2 replicas. It handles card data and has to meet the strictest standard. |
A few minutes ago, the platform team added a Pod Security label to the monitoring namespace, and node-agent stopped starting Pods right away. It is down on every node. storefront and payments have no Pod Security labels yet.
Work from cplane-01. The API server on that machine writes its audit log to /var/log/kubernetes/audit.log, which needs sudo to read.
Task
- Get the
node-agentDaemonSet running on every node. Its host access is legitimate, so change themonitoringnamespace to the level that allows it, and leave the DaemonSet as it is. - Set up the
storefrontnamespace for a staged rollout: enforce thebaselinelevel now, and report anything that would break therestrictedlevel, both to whoever applies a change and in the API server audit log. Thestorefront-webDeployment must keep running. - Make the
paymentsnamespace enforce therestrictedlevel, with thepayments-apiDeployment running2replicas that all meet that level.
Set every policy with the Pod Security Admission labels on the namespace. For the payments-api Deployment, change its Pod spec, not its image, and do not add exemptions. Leave the node-agent DaemonSet and the storefront-web Deployment as they are.
Hint 1 | Finding out why a controller creates no Pods
When an admission controller rejects a Pod, the controller that tried to create it records the reason as an event:
kubectl describe daemonset <name> -n <namespace>
kubectl get events -n <namespace> --field-selector reason=FailedCreate
kubectl get namespace <namespace> --show-labels
Hint 2 | The shape of a Pod Security label
Each namespace label sets one mode to one level. A namespace can carry one label per mode:
pod-security.kubernetes.io/<MODE>=<LEVEL>
kubectl label namespace <namespace> <label>=<value> --overwrite
A server-side dry run shows which running Pods a new enforce level would reject, without changing anything:
kubectl label --dry-run=server --overwrite namespace <namespace> <label>=<value>
Hint 3 | When a running Deployment is not compliant
A new enforce level never touches Pods that are already running, so a Deployment can look healthy while every new Pod is rejected. Force a rollout and see what the ReplicaSet reports:
kubectl rollout restart deployment/<name> -n <namespace>
kubectl get events -n <namespace> --field-selector reason=FailedCreate
The rejection message names each field the Pod spec needs, at Pod or container level:
securityContext:
<field>: <value>
Hint 4 | Seeing what audit mode recorded
Audit mode does not print anything. It adds an annotation to the matching entry in the API server audit log:
sudo grep -c <annotation> /var/log/kubernetes/audit.log