Kubernetes Security - Intermediate

Setup the environment

Click on the START PLAYGROUND button
Wait for the playground to start
Clone the repository

git clone https://github.com/Alevsk/dvka.git ~/dvka
cd ~/dvka/workshop

Run install-tools.sh script and follow the instruction

sudo ./install-tools.sh --install

After that you can start the intermediate labs by going into each lab directory and follow the instructions there, e.g. cd labs/service-account-token.

1 Kubernetes Certificate Authority

Prerequisites

A running Kubernetes cluster.
kubectl installed and configured to connect to your cluster.

Quick Start

Generate a Private Key
"Elliptic Curve Digital Signature Algorithm" (ECDSA) with the P-256 curve. ECDSA is a widely-used and secure algorithm for generating key pairs.
```
openssl ecparam -name prime256v1 -genkey -noout -out private.key
```
Optionally you could use a different algorithm, ie: RSA.
```
openssl genrsa -out private.key 2048
```
Generate a Certificate Signing Request (CSR)
```
openssl req -new -config cert.cnf -key private.key -out kubernetes-security.csr
```
Note: Use https://www.sslshopper.com/csr-decoder.html to verify the generated csr
Encode the CSR in Base64 and copy it to your clipboard
```
cat kubernetes-security.csr | base64 | tr -d "\n"
```
Open the csr.yaml file and paste the encoded certificate on the spec.request field if is not there already
Create the CSR resource in Kubernetes
```
kubectl apply -f csr.yaml
```
Using k9s or kubectl list and inspect the created CSR resource
```
kubectl get csr -A
```

Manually approve the CSR

kubectl certificate approve kubernetes-security-csr

Retrieve the Signed Certificate
```
kubectl get csr kubernetes-security-csr -o jsonpath='{.status.certificate}'| base64 -d > public.crt
```
When the CSR is approved, the new certificate will be issued by the Kubernetes CA and would be found under the status.certificate field of the kubernetes-security-csr csr
Verify the public.crt certificate was issued by kubernetes using https://www.sslchecker.com/certdecoder or the openssl command
```
cat public.crt | openssl x509 -noout -text
```
Use this keypair to configure TLS for your workloads, similar to what you did for the Configmaps & Secrets lab

Resources

Certificate Signing Requests

2 cert-manager: X.509 Certificate Management for Kubernetes

Prerequisites

A running Kubernetes cluster.
kubectl installed and configured to connect to your cluster.

Quick Start

Install cert-manager using the default yaml configuration
```
kubectl apply -f cert-manager.yaml
```

Provide or generate your own root Certificate Authority (CA), ie:

# generate private key
openssl genrsa -out rootCAKey.pem 2048
# generate public key
openssl req -x509 -sha256 -new -nodes -key rootCAKey.pem -days 3650 -out rootCACert.pem

base64 encode the public and private keys for your root CA

cat rootCACert.pem | base64 -w 0 # copy to clipboard
cat rootCAKey.pem | base64 -w 0 # copy to clipboard

Open the custom-ca-secret.yaml file and place the base64 encoded values for the public and private key in the tls.crt and tls.key fields, then create create the custom ca on kubernetes
```
# create secret
kubectl apply -f custom-ca-secret.yaml
# create cert-manager ClusterIssuer
kubectl apply -f custom-ca.yaml
```
Verify the ca was added to the cluster issuers list
```
kubectl get clusterissuers
```
Generate a new tls certificate using cert-manager and your root CA via cert-manager
```
kubectl apply -f default-tls-certificate.yaml
```

Verify the certificate was generated correctly

# check status of the certificate request
kubectl get certificaterequests
# check status of the certificate itself
kubectl get certificates
# check the tls certificate stored directly on the k8s secret
kubectl describe secrets default-tls-certificate-secret describe
# inspect the public and private keys of the generated certificate
kubectl get secrets default-tls-certificate-secret -o yaml

You can use https://www.sslshopper.com/certificate-decoder.html or https://www.sslchecker.com/certdecoder to verify the content of the certificate

Use this keypair to configure TLS for your workloads, similar to what you did for the Configmaps & Secrets lab

End the lab

kubectl delete secret default-tls-certificate-secret
kubectl delete -f default-tls-certificate.yaml
kubectl delete -f custom-ca-secret.yaml
kubectl delete -f custom-ca.yaml
kubectl delete -f cert-manager.yaml

Resources

3 Pod Resource Limits

Prerequisites

A running Kubernetes cluster.
kubectl installed and configured to connect to your cluster.

Quick Start

Install metrics server in your cluster
```
kubectl apply -f metrics-server.yaml
```
After the service is running restart k9s and look for the new CPU and Memory metrics available for your cluster. Look at metrics using kubectl:
```
kubectl top pod -A
```

Memory testing

# deploy container without limits
kubectl apply -f mem-testing.yaml
# look at memory consumtion by the pod
watch kubectl top pod -l app=mem-testing

Update memory stress container to limit the amount of memory to only 100mb

# deploy container with limits
kubectl apply -f mem-testing-limits.yaml

Observe how the container is stopped with status OOMKilled

CPU testing
```
# deploy container without limits
kubectl apply -f cpu-testing.yaml
# look at cpu consumtion by the pod
watch kubectl top pod -l app=cpu-testing
```
The -cpus "2" argument tells the Container to attempt to use 2 CPUs.

Update cpu stress container to limit the amount of cpu to only 1 core
```
# deploy container with limits
kubectl apply -f cpu-testing-limits.yaml
```
Observe how the container is limited to consume maximum 1 cpu

Finalize the lab

kubectl delete -f metrics-server.yaml
kubectl delete -f mem-testing.yaml
kubectl delete -f cpu-testing.yaml

Resources

Assign Memory Resources to Containers and Pods

4 Scratch Containers

Prerequisites

A running Kubernetes cluster.
kubectl installed and configured to connect to your cluster.
Docker installed locally.
Go installed locally.

Quick Start

Take a look at the example application source code under ./encoding-service
- main.go
- base64
- Dockerfile

Build the encoding service docker image

docker build -t alevsk/dvka:lab10 -f encoding-service/Dockerfile ./encoding-service

Push the image to your Kubernetes cluster

kind load docker-image alevsk/dvka:lab10 --name workshop-cluster

Deploy the application into Kubernetes

# create deployment
kubectl apply -f encoding-service.yaml
# locally expose the application service
kubectl port-forward svc/encoding-service 1337:1337

Open the browser and go to http://localhost:1337/run?command=encode&message=hello%20world
- Found any vulnerabilities?
- Get a shell on the container using kubectl or k9s
Stop port-forward (<ctrl+c>) and remove application
```
kubectl delete -f encoding-service.yaml
```

Build the scratch container and deploy to kubernetes again

# build scratch image
docker build -t alevsk/dvka:lab10-scratch -f encoding-service/Dockerfile.scratch ./encoding-service
# push image to kubernetes
kind load docker-image alevsk/dvka:lab10-scratch --name workshop-cluster
# create deployment
kubectl apply -f encoding-service-scratch.yaml
# locally expose the application service
kubectl port-forward svc/encoding-service 1337:1337

Open the browser and go to http://localhost:1337/run?command=encode&message=hello%20world
- Test for vulnerabilities again
- Get a shell on the container using kubectl or k9s
Stop port-forward (<ctrl+c>) and remove application
```
kubectl delete -f encoding-service.yaml
```
Follow up
- Differences between regular images and scratch images

Resources

5 Service Account Tokens

Prerequisites

A running Kubernetes cluster.
kubectl installed and configured to connect to your cluster.

Quick Start

Deploy ubuntu pod

# create ubuntu pod
kubectl apply -f ubuntu.yaml

Exec into the running container
kubectl:
```
kubectl exec -it pod/ubuntu -- /bin/bash
```
k9s:
Pods > ubuntu > press <s>
Install curl
- apt-get update && apt-get install curl jq -y
Move to the serviceaccount folder
- cd /var/run/secrets/kubernetes.io/serviceaccount
Analyze the 3 files under the serviceaccount folder
- ca.crt
- namespace
- token
Visualize token using https://jwt.io/ or a similar tool

Query the the Kubernetes api server

curl https://kubernetes.default.svc.cluster.local
# ignore tls verification
curl https://kubernetes.default.svc.cluster.local -k
# pass ca.crt to verify tls connection
curl https://kubernetes.default.svc.cluster.local --cacert ca.crt

Authenticate using the service account token

export TOKEN=$(cat token)
curl --cacert ca.crt https://kubernetes.default.svc.cluster.local/api/v1/namespaces?limit=500 -H "Authorization: Bearer $TOKEN"
# Use jq to parse the list of existing namespaces in the cluster
curl --cacert ca.crt https://kubernetes.default.svc.cluster.local/api/v1/namespaces?limit=500 -H "Authorization: Bearer $TOKEN" | jq ".items[].metadata.name"

Deploy ubuntu pod

# delete ubuntu pod
kubectl delete -f ubuntu.yaml
# create ubuntu pod without mounting service account by default
kubectl apply -f ubuntu-no-sa.yaml

Try to navigate again to the serviceaccount folder (You should get an error)
- kubectl exec -it pod/ubuntu -- /bin/bash
- cd /var/run/secrets/kubernetes.io/serviceaccount

Finalize the lab

# end the lab
kubectl delete -f ubuntu-no-sa.yaml
kubectl delete -f ubuntu.yaml

Resources

Configure Service Accounts for Pods

6 Network Policies with Calico

Prerequisites

A running Kubernetes cluster.
kubectl installed and configured to connect to your cluster.

Quick Start

Look at tenant-1.yaml file and deploy all the resources for application 1
```
kubectl apply -f tenant-1.yaml
```
Look at tenant-2.yaml file and deploy all the resources for application 2
```
kubectl apply -f tenant-2.yaml
```

Inspect the resources created for the tenant-1 and tenant-2 namespaces using k9s or kubectl

# tenant-1
kubectl get all --namespace tenant-1
# tenant-2
kubectl get all --namespace tenant-2

Exec into the running container
kubectl:

# exec into nginx tenant-1
kubectl -n tenant-1 exec -it <pod name> -- sh
# exec into nginx tenant-2
kubectl -n tenant-2 exec -it <pod name> -- sh

k9s:

Namespace > tenant-1 > Pods > nginx > press <s>
Namespace > tenant-2 > Pods > nginx > press <s>

Install curl on both nginx containers
- apk add curl
Test connectivity between services in two different namespaces
From tenant-1 to tenant-2
```
curl http://nginx.tenant-2.svc.cluster.local:8080
```
From tenant-2 to tenant-1
```
curl http://nginx.tenant-1.svc.cluster.local:8080
```
Notice how the service URLs have the following structure: http://<service name>.<namespace>.svc.cluster.local:<port>

Install the Tigera Calico operator and custom resource definitions.

kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/tigera-operator.yaml

Install Calico by creating the necessary custom resource.
Before creating this manifest, read its contents and make sure its settings are correct for your environment. For example, you may need to change the default IP pool CIDR to match your pod network CIDR. See https://docs.tigera.io/calico/latest/getting-started/kubernetes/quickstart.
```
kubectl create -f custom-resources.yaml
```
Confirm that all of the pods are running with the following command.
```
watch kubectl get pods -n calico-system
```
Block all incomming request to tenant-2 namespace workloads using Calico Networking Policies. Look at np-default-deny.yaml and then run:
Note: In some cluster setups nginx pods need to be restarted first.
```
kubectl apply -f np-default-deny.yaml
```
Exec into the tenant-1 running container again and test connectivity to the running container in tenant-2. You should see a timeout error.
```
curl -m 5 http://nginx.tenant-2.svc.cluster.local:8080
```

Exec into the tenant-2 running container again and test connectivity to the running container in tenant-1.

# request to tenant-1
curl -m 5 http://nginx.tenant-1.svc.cluster.local:8080
# request to itself using service name should timeout
curl -m 5 http://nginx.tenant-2.svc.cluster.local:8080
# request to itself using localhost
curl -m 5 http://localhost:8080

Deploy networking rule to allow internal connectivity for tenant-2 namespace.

kubectl apply -f np-allow-namespace-connectivity.yaml

Exec into the tenant-2 running container again and test connectivity

# request to itself using service name should work this time
curl -m 5 http://nginx.tenant-2.svc.cluster.local:8080

Deploy a new tenant namespace. Update existing networking rule to allow connectivity from workloads running on tenant-3 namespace to tenant-2 namespace.

# deploy tenant-3
kubectl apply -f tenant-3.yaml
# update tenant-2 networking rule to accept tenant-3 connections
kubectl apply -f np-allow-namespace-connectivity-update.yaml

Exec into the tenant-3 running container
kubectl:

# exec into nginx tenant-3
kubectl -n tenant-3 exec -it <pod name> -- sh

k9s:

Namespace > tenant-3 > Pods > nginx > press <s>

Install curl on tenant-3 container and test connectivity to tenant-2.

# install curl
apk add curl
# request to tenant-2 should work this time
curl -m 5 http://nginx.tenant-2.svc.cluster.local:8080

Finalize the lab

# end the lab
kubectl delete -f tenant-1.yaml
kubectl delete -f tenant-2.yaml
kubectl delete -f tenant-3.yaml
kubectl delete -f np-default-deny.yaml
kubectl delete -f np-allow-namespace-connectivity.yaml
kubectl delete -f np-allow-namespace-connectivity-update.yaml
kubectl delete -f custom-resources.yaml
kubectl delete -f https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/tigera-operator.yaml