Namespaces, RBAC and ServiceAccounts

Until now you have worked as the absolute administrator of a single-tenant cluster. Real clusters are not like that: they are shared by teams, applications and environments that must not step on each other. The two tools to keep that in order are Namespaces, which divide the cluster into logical spaces, and RBAC, which decides who can do what in each one.

The book's Security chapter carefully separates authentication (who you are) from authorization (what you can do). This module deals with the second, which is the one you touch with your hands.

The goal of this unit: give team A's space an identity that can look at its Pods but not touch them. Work from the dev-machine tab.

Step 1: The Namespace you have been working in

Before creating anything, a check that is going to explain the whole course backwards:

kubectl config view --minify -o jsonpath='{..namespace}'; echo

kubectl get namespaces

You have spent the whole course working inside a Namespace and had not noticed. You never wrote -n because the context already pointed at tienda from the first lesson. That is exactly what a well-placed Namespace does: disappear.

Now that you look at it head-on, things you saw in passing fall into place. The internal DNS in the Exposing the application with a Service lesson resolved web.tienda.svc.cluster.local, and that second piece was not decoration: it was this. And in the DaemonSet lesson the agent had to move to plataforma because tienda would not accept its hostPath, which is the first time a Namespace said no to you.

Notice also the ones that are there and are not yours: kube-system (the cluster's components, including the Traefik from the Networking module), default (the one Kubernetes always creates, and where the API server's own Service lives) and plataforma.

Create the neighboring team's, which in a while will serve to check how far the permissions you grant reach:

kubectl create namespace equipo-b

kubectl get namespaces

Step 2: The identity (ServiceAccount)

RBAC authorizes identities, so first you need one. ServiceAccounts are the identities of workloads (the processes that run in Pods) and they also work perfectly for practicing RBAC:

kubectl create serviceaccount agent-sa -n tienda

Note the -n tienda: the ServiceAccount, like almost everything from now on, lives inside the namespace.

And that is no exaggeration: check it with the star command of this lesson, which answers permission questions without having to try them:

kubectl auth can-i list pods --as=system:serviceaccount:tienda:agent-sa -n tienda

It answers no. In RBAC everything is denied by default; permissions only add up.

Step 3: The permissions (Role and RoleBinding)

RBAC works with two pieces that people constantly confuse: the Role defines a set of permissions, and the RoleBinding hands them to an identity. A Role without a binding is an unsigned paper. Create rbac.yaml with both:

cat << 'EOF' > rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: lector-pods
  namespace: tienda
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: agent-lector-pods
  namespace: tienda
subjects:
- kind: ServiceAccount
  name: agent-sa
  namespace: tienda
roleRef:
  kind: Role
  name: lector-pods
  apiGroup: rbac.authorization.k8s.io
EOF

The YAML, explained in questions and answers

What does the --- between the two objects do?

It is YAML's document separator: one file can contain several objects and kubectl apply -f creates them all. Very common for pieces that only make sense together, like these two.

Why apiGroups: [""]?

The empty string designates the core group of the API, the one Pods belong to. If the Role granted permissions over Deployments, it would say ["apps"] here. It is the same division into groups you have been seeing in the apiVersion of the whole course.

What exactly are the verbs?

The actions of the Kubernetes API: the reading triad is get (one), list (all) and watch (subscribe to changes). The writing ones (create, update, patch, delete) are deliberately left out of this Role.

What does the RoleBinding join with what?

subjects (to whom: the ServiceAccount agent-sa) with roleRef (what: the Role lector-pods, "pod reader"). The subjects can also be users or groups, and the same Role can be handed out to many subjects with different bindings.

And if I wanted to grant these permissions across the whole cluster?

There are the namespace-less versions: ClusterRole and ClusterRoleBinding. The mechanics are identical; the scope, global. Always start with the namespaced version: in security, the small scope is the right scope.

Apply it and repeat the interrogation, now with the two questions that define the goal of the lesson:

kubectl apply -f rbac.yaml

kubectl auth can-i list pods --as=system:serviceaccount:tienda:agent-sa -n tienda

kubectl auth can-i delete pods --as=system:serviceaccount:tienda:agent-sa -n tienda

yes and no. Exactly the least privilege that was asked for. And now the question that closes the lesson, the same as before but pointing at the neighboring team's namespace:

kubectl auth can-i list pods --as=system:serviceaccount:tienda:agent-sa -n equipo-b

Also no, and that is the difference between a Role and a ClusterRole: the permission you just granted does not exist outside tienda. That is why equipo-b was there from the start.

Summary

• Namespaces divide the cluster into logical spaces with their own content and their own rules.
• RBAC denies everything by default; permissions are granted with Role (what) plus RoleBinding (to whom).
• kubectl auth can-i --as=... answers permission questions without running anything: use it before, during and after every RBAC change.
• ClusterRole and ClusterRoleBinding are the global-scope variants, for when they are really needed.

The namespace now has identities and permissions. In the next unit you will see what a ServiceAccount is really for: giving it to a Pod.

Until now you have used the ServiceAccount agent-sa as a stand-in: an identity to practice RBAC with from your terminal, with --as=. But its real job is another. A ServiceAccount is the identity with which a process running inside a Pod talks to the Kubernetes API.

And here is the fact that wakes a lot of people up: every Pod you have created in this course already had one. Check it:

kubectl get pods -n tienda -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.serviceAccountName}{"\n"}{end}'

They all use the default ServiceAccount, which exists automatically in every namespace. If your Pod never talks to the API, that identity is of no use to it at all, and it carries it anyway.

Step 1: Give a Pod the right identity

Create agent.yaml, a Pod that does need to look at the cluster:

cat << 'EOF' > agent.yaml
apiVersion: v1
kind: Pod
metadata:
  name: agent
  namespace: tienda
spec:
  serviceAccountName: agent-sa
  containers:
  - name: app
    image: ghcr.io/iximiuz/labs/nginx:alpine
    command: ["sh", "-c", "sleep infinity"]
EOF

A single new line, serviceAccountName, and that Pod stops being anonymous.

kubectl apply -f agent.yaml
kubectl wait --for=condition=Ready pod/agent -n tienda --timeout=60s
kubectl exec agent -n tienda -- ls -l /var/run/secrets/kubernetes.io/serviceaccount/

Three files show up inside the container without anyone asking for them:

• token: a JWT signed by the cluster. It is the credential.
• ca.crt: the cluster's certificate authority, so the Pod can verify it is talking to the real API server.
• namespace: the Pod's namespace, for convenience.

The questions worth asking

Where does that token come from, if nobody created any Secret?

From a projected volume. The kubelet asks the API server for a short-lived token, writes it to the container's filesystem and renews it on its own before it expires. It used to be done another way: each ServiceAccount had a Secret with an eternal token stored in etcd. A token that never expires and sits in a Secret that anyone with read permission can read is exactly what it sounds like, so it was changed. It is the same mechanism behind the kubectl create token you will use in the Working with the cluster module.

What can this Pod do with its token?

Exactly what its RBAC allows, no more, no less: list Pods in tienda, because that is what you granted it with the RoleBinding. Not delete, not leave its namespace. The identity is worth nothing on its own; what counts are the permissions you have tied to it.

Check it from inside the Pod itself, talking to the API raw:

kubectl exec agent -n tienda -- sh -c '
  TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
  curl -s --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
    -H "Authorization: Bearer ${TOKEN}" \
    https://kubernetes.default.svc/api/v1/namespaces/tienda/pods' | jq .

It answers with the list of Pods in JSON. The three files of the volume come into play at once: the token goes in the header, the ca.crt verifies that the real API server is on the other side, and the namespace is the one you wrote in the URL.

And now ask it for something it is not entitled to, the Pods in the default namespace:

kubectl exec agent -n tienda -- sh -c '
  TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
  curl -s --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
    -H "Authorization: Bearer ${TOKEN}" \
    https://kubernetes.default.svc/api/v1/namespaces/default/pods' | jq .

The API answers a Failure with the reason spelled out in full: pods is forbidden: User "system:serviceaccount:tienda:agent-sa" cannot list resource "pods". RBAC works the same whether the request comes from your terminal or from inside a container, and the subject that shows up in the error is exactly the same one you have been writing in the --as=. Notice also the name kubernetes.default.svc: it is an ordinary Service in the default namespace, with its ClusterIP, exactly like the ones you created in the Networking module. The API server is discovered through the internal DNS, like everything else.

Step 2: The Pod that does not want an identity

And now the question almost nobody asks: why does a Pod that is never going to talk to the API carry a token? Your nginx does not need it. It is a credential mounted inside a container that serves web pages to the internet, waiting for someone to find a vulnerability and read it.

Create api.yaml:

cat << 'EOF' > api.yaml
apiVersion: v1
kind: Pod
metadata:
  name: api
  namespace: tienda
spec:
  automountServiceAccountToken: false
  containers:
  - name: app
    image: ghcr.io/iximiuz/labs/nginx:alpine
    command: ["sh", "-c", "sleep infinity"]
EOF

kubectl apply -f api.yaml
kubectl wait --for=condition=Ready pod/api -n tienda --timeout=60s

The check below prints in Spanish: SÍ hay directorio de token means "YES, there is a token directory", and NO existe means "does NOT exist".

kubectl exec api -n tienda -- sh -c '
  if [ -d /var/run/secrets/kubernetes.io/serviceaccount ]; then
    echo "SÍ hay directorio de token:"
    ls /var/run/secrets/kubernetes.io/serviceaccount/
  else
    echo "NO existe /var/run/secrets/kubernetes.io/serviceaccount"
  fi
'

NO existe. And it is not that the directory is empty: it never gets created. Without the projected volume, Kubernetes mounts nothing under /var/run/secrets, so in there not even the folder exists.

Run the same command against the earlier Pod, to see the difference side by side:

kubectl exec agent -n tienda -- sh -c '
  if [ -d /var/run/secrets/kubernetes.io/serviceaccount ]; then
    echo "SÍ hay directorio de token:"
    ls /var/run/secrets/kubernetes.io/serviceaccount/
  else
    echo "NO existe /var/run/secrets/kubernetes.io/serviceaccount"
  fi
'

SÍ, with its three files. Same command, same cluster, same image: the only difference is one line of the manifest. No token, no credential to steal.

Where does that field go? On the Pod, as here, or directly on the ServiceAccount (automountServiceAccountToken: false in its metadata), and then it affects every Pod that uses it. Setting it on a namespace's default ServiceAccount is one of the cheapest and most effective hardening measures there are: nobody receives a token by accident, and whoever needs one will have to ask for it explicitly.

Summary

• A ServiceAccount is a Pod's identity before the API. Every Pod carries one, even if it does not use it.
• The token is mounted as a projected volume, expires and renews on its own. It no longer lives in an eternal Secret.
• What a Pod can do with its token is decided by RBAC, exactly as with a user.
• automountServiceAccountToken: false on the Pods (or on the ServiceAccount itself) for everything that does not talk to the API.