This lesson was created by a community author. Community content is reviewed by the iximiuz Labs team on a best effort basis.

Lesson in Software Bill of Behavior - vendor supplied runtime profile for tampering and anomaly detection

The agent suggests the client's SBOB — you sign it

Node-Agent can now propose SBOB overlays. Compose them, sign them, bind them, and tamper with them

Suggest, prove, sign, bind

While the SBOB itself is the simplest possible runtime profile that captures contrast between benign and malicious behavior, and thus needs several iterations of tuning, node-agent can discover the extensions.

The most common case is that an SBOB contains the topology of a deployment's implementation details, that is the internals of that deployement.
However, when you connect other components (ingress, clients, backends, queues etc), you need a way to compose. The auto-discovery of these connections, via protocols or sockets, mounts or otherwise must have good UX. This is why I made this lab, so you can try it and suggest improvements.
Use-cases are also welcome.

Note

Features in this module are not GA and subject to change at any time

0. One terminal for logs

Note

If you dont care for the manual step through of unit-1, feel free to skip to unit-2

Split screen for logs

Open the split screen and a second tab logs, to watch the node-agent logs

kubectl -n honey logs ds/node-agent -c node-agent --all-pods=true -f | sed -u 's/^\[[^]]*\] //' | jq -Rc --unbuffered 'fromjson? | if .RuleID then {t:.time, rule:.RuleID, alert:.BaseRuntimeMetadata.alertName, ns:.RuntimeK8sDetails.namespace, wl:.RuntimeK8sDetails.workloadName, msg:.message} else {t:.ts, level, msg} end'

{"t":"…","level":"info","msg":"suggester enabled"}
{"t":"…","level":"info","msg":"Started tracer"}
{"t":"…","level":"info","msg":"ContainerWatcher started successfully"}
{"t":".","level":"info","msg":"RogueArtifact writer wired: discoverable signoff state enabled"}
{"t":".","level":"info","msg":"queue processor started"}
{"t":".","level":"info","msg":"Successfully verified object signature"}
{"t":".","level":"info","msg":"signed bundle overlays enabled in alert mode"}
{"t":".","level":"warn","msg":"signed bundle overlays anchored to the PUBLISHED DEMO root key: this authenticates nothing, mount a real root before relying on signatures"}
{"t":".","level":"warn","msg":"rule signing DISABLED: the trust policy in force carries no ruleClasses; ANY Rules object in ANY namespace will load without a signature check"}
{"t":".","level":"info","msg":"trust policy in force"}

1. The vendor ships redis with a signed SBOB — no external dependecies

_redis with SBOB signed by vendor _

Deploy a redis-DB with a signed SBOB

cd ~/bob/example/redis
export CP=containerprofiles.spdx.softwarecomposition.kubescape.io
kubectl create ns redis
kubectl apply -f vendor/redis-base-signed.yaml
helm install redis oci://registry-1.docker.io/bitnamicharts/redis --version 27.0.18 -n redis --set architecture=standalone --set auth.enabled=false --set image.repository=bitnami/redis --set image.digest=sha256:08863c2c3f4e051fb6139b38fa223e9c13be5033326a59bead182860d899bf98 \
  --set 'master.podLabels.kubescape\.io/user-defined-profile=redis'

{
  "level": "warn",
  "msg": "signed fragment stored spec diverges from the signed content:   
          the stored spec is display-only and is NOT enforced; enforcement uses the signed content"
}
{"level":"info","msg":"adopted user-authored ContainerProfile as authoritative base"}
{"level":"info","msg":"container has a user defined profile"}
{"level":"info","msg":"Successfully verified object signature"}

The reason for the tamper alert is that during the first tick, the operator hasnt reconciled

the vendor's SBOB, abridged — 15 execs, 79 opens, 8 capabilities · NO ingress

kind: ContainerProfile
metadata:
  name: redis
  namespace: redis
  annotations:
    kubescape.io/managed-by: User
    signature.kubescape.io/identity: local-key
    signature.kubescape.io/issuer: local
    signature.kubescape.io/timestamp: "1790366987"
    signature.kubescape.io/signature: MEUCIQDd5Ba9Hg2iSh2OAO8w…
    signature.kubescape.io/certificate: LS0tLS1CRUdJTiBDRVJUSUZJ…
    signature.kubescape.io/content: H4sIAAAAAAAA/7xZ73LiOBJ/…
spec:
  matchLabels: {app.kubernetes.io/component: master, app.kubernetes.io/instance: redis, app.kubernetes.io/name: redis}
  capabilities: [CAP_CHOWN, CAP_DAC_OVERRIDE, CAP_NET_ADMIN, CAP_SETGID, CAP_SETUID, CAP_SYS_ADMIN, …]
  execs:
    - {args: [/opt/bitnami/redis/bin/redis-cli, -h, localhost, -p, "6379", ping], path: /opt/bitnami/redis/bin/redis-cli}
    - {args: [/opt/bitnami/redis/bin/redis-server, ⋯⋯], path: /opt/bitnami/redis/bin/redis-server}
    - {args: [/health/ping_liveness_local.sh, ⋯⋯], path: /usr/bin/bash}
    - …
  opens:
    - {flags: [O_RDONLY], path: /data}
    - {flags: [O_APPEND, O_CREAT, O_WRONLY], path: /data/appendonlydir/appendonly.aof.1.incr.aof}
    - …
  ingress: null

Observe, that a database vendor cannot know external dependencies, which is why the SBOB ships without external network connections:

  ingress: null
  egress:  null

2. A rogue client appears

The new client is a rogue artefact

Deploy a redis-client pod, that carries no SBOB:

sed '/kubescape.io\/user-defined-profile/d' client.yaml | kubectl apply -f -
kubectl -n redis rollout status deploy/redis-client --timeout=120s

Alerts R1017 Rogue artefact, R0012 Unexpected Ingress Network Traffic on redis for every connection, R0040 Unexpected process arguments

{
  "rule": "R1017",
  "alert": "Rogue artefact",
  "ns": "redis",
  "wl": "redis-client",
  "msg": "Container 'client' in namespace 'redis' has no bound profile — an unmanaged/unknown container is running in the cluster"
}
{
  "rule": "R0012",
  "alert": "Unexpected Ingress Network Traffic",
  "ns": "redis",
  "wl": "redis-master",
  "msg": "Unexpected ingress network communication from: 10.42.0.49:6379 using TCP to: redis"
}
{
  "rule": "R0040",
  "alert": "Unexpected process arguments",
  "ns": "redis",
  "wl": "redis-client",
  "msg": "Unexpected process arguments: redis-cli with PID 76287"
}

You are now seeing every single alert on the redis-db, which is considered trusted, whereas the client traffic is not.

Important

If you are on production, the R1017 is an absolute 🆘 code-red alert. This exercise of discovering the topology should be done in a lower environment.

If you have workloads that spawn pods, those should carry SBOBs that include the intended wildcard for their maximal allowlists. Admission controllers can achieve this

3. The agent suggests a profile (~2 min)

The rogue client is being profiled now by a shadow_profile, which cannot be deleted.

The configuration settings are such that alerts are not deduplicated (no Cooldown) until you heal the rogue object. The agent's helper is suggesting you the most likely profile at any tick (30s) after a minimum time (2min).

You can skip the waiting

If you dont want to wait, the profile is already prepared for you under: yolo/client-sbob.json

Else:

watch -n5 "kubectl -n redis get $CP -l kubescape.io/profile-role=suggestion"

kubectl -n redis get events --field-selector reason=SBoBSuggested -o custom-columns=POD:.involvedObject.name,MSG:.message

POD            MSG
redis-client   SBoB suggestion replicaset-redis-client-client-097b-1726-suggested ready. Accept with: kubectl -n redis label pod redis-client kubescape.io/user-defined-profile=replicaset-redis-client-client-097b-1726-suggested

Do not follow the suggestion, as we want to sign the profile. At this point you should inspect the suggestion and correct as needed.

discovered vs. portable

The shadow being learned

suggestion — for you to review

125 execs · 13 opens · 63 syscalls · 2 egress

spec:
  execs:
    - {path: /bin/busybox, args: [/bin/sh, -c, "i=0\nwhile true; do\n  redis-cli -h redis-master -p 6379 SET \"k:$i\" \"v:$i\" >/dev/null\n  i=$((i+1)); sleep 2\ndone\n"]}
    - {path: /bin/busybox, args: [sleep, "2"]}
    - {path: /pause, args: [/pause]}
    - {path: /usr/local/bin/redis-cli, args: [/usr/local/bin/redis-cli, -h, redis-master, -p, "6379", SET, k:32, v:32]}
    - {path: /usr/local/bin/redis-cli, args: [/usr/local/bin/redis-cli, -h, redis-master, -p, "6379", SET, k:33, v:33]}
    - {path: /usr/local/bin/redis-cli, args: [/usr/local/bin/redis-cli, -h, redis-master, -p, "6379", SET, k:34, v:34]}
    - … 119 more redis-cli lines, one per SET
  opens:
    - {path: /etc/resolv.conf, flags: [O_CLOEXEC, O_LARGEFILE, O_RDONLY]}
    - {path: /etc/hosts, flags: [O_CLOEXEC, O_LARGEFILE, O_RDONLY]}
    - {path: /lib/libssl.so.3, flags: [O_CLOEXEC, O_LARGEFILE, O_RDONLY]}
    - {path: /proc/sys/vm/overcommit_memory, flags: [O_CLOEXEC, O_RDONLY]}
    - … 9 more
  syscalls: [accept4, bind, brk, clone, close, connect, dup2, execve, exit_group, fcntl, getpid, ioctl, … 51 more]
  egress:
    - identifier: f8ed3623…
      type: internal
      serviceRefName: redis-master
      serviceRefNamespace: redis
      podSelector:
        matchLabels: {app.kubernetes.io/component: master, app.kubernetes.io/instance: redis, app.kubernetes.io/name: redis}
      ports: [{name: TCP-6379, port: 6379, protocol: TCP}]
    - identifier: f14f2fcf…
      type: internal
      serviceRefName: kube-dns
      serviceRefNamespace: kube-system
      namespaceSelector:
        matchLabels: {kubernetes.io/metadata.name: kube-system}
      podSelector:
        matchLabels: {k8s-app: kube-dns}
      ports: [{name: UDP-53, port: 53, protocol: UDP}]

Note, that the execs are collapsed very generously and the syscalls removed. The suggestion still keep the redis-port set at 6397, so if you intend to run the client differently in your production env, you must correct that.

4. You sign it with your key and bind it

Validate and Sign off the suggestion

yolo/client-sbob.json is exactly what the agent suggests for this client. Use it instead of the first three lines below:

cp yolo/client-sbob.json client-sbob.json

If you waited, use this command:

~/sign-object generate-keypair --output operator.pem
N=$(kubectl -n redis get $CP -l kubescape.io/profile-role=suggestion -o jsonpath='{.items[0].metadata.name}')
kubectl -n redis get $CP "$N" -o json | python3 -c '
import sys,json
d=json.load(sys.stdin)
d["metadata"]={"name":"redis-client","namespace":"redis","annotations":{"kubescape.io/managed-by":"User"}}
json.dump(d,sys.stdout)' > client-sbob.json
~/bobctl validate --file client-sbob.json
~/sign-object sign --file client-sbob.json --key operator.pem --output client-sbob-signed.yaml
kubectl apply -f client-sbob-signed.yaml
kubectl -n redis patch deploy redis-client -p '{"spec":{"template":{"metadata":{"labels":{"kubescape.io/user-defined-profile":"redis-client"}}}}}'
kubectl -n redis rollout status deploy/redis-client --timeout=120s

~/bobctl validate --file client-sbob.json
~/sign-object sign --file client-sbob.json --key operator.pem --output client-sbob-signed.yaml

✓ Private key written to: operator.pem
✓ Public key written to: operator.pem.pub
OK: profile "client-sbob.json" is valid (5 execs, 13 opens, 0 syscalls)
[info] Successfully signed object. namespace: redis; name: redis-client; identity: local-key; issuer: local
✓ Profile signed successfully
  Issuer: local
  Identity: local-key
  Timestamp: 1790456943
✓ Signed profile written to: client-sbob-signed.yaml
containerprofile.spdx.softwarecomposition.kubescape.io/redis-client created
deployment.apps/redis-client patched

{"t":"…","level":"info","msg":"Successfully verified object signature"}
{"t":"…","level":"info","msg":"adopted user-authored ContainerProfile as authoritative base"}

You may notice that not all alerts disappear. Reason is that we only healed one half of the network connection.

client-sbob-signed.yaml, abridged

kind: ContainerProfile
metadata:
  name: redis-client
  namespace: redis
  annotations:
    kubescape.io/managed-by: User
    signature.kubescape.io/identity: local-key
    signature.kubescape.io/issuer: local
    signature.kubescape.io/signature: MEUCIQClx5bZ…
    signature.kubescape.io/content: H4sIAAAAAAAA/7RW…
spec:
  architectures: [amd64]
  matchLabels: {app: redis-client}
  execs:
    - {args: [/bin/sh, ⋯⋯], path: /bin/busybox}
    - {args: [/usr/local/bin/redis-cli, -h, redis-master, -p, "6379", SET, ⋯, ⋯], path: /usr/local/bin/redis-cli}
    - …
  egress:
    - {serviceRefName: redis-master, serviceRefNamespace: redis, ports: [{name: TCP-6379, port: 6379, protocol: TCP}], …}
    - {serviceRefName: kube-dns, serviceRefNamespace: kube-system, ports: [{name: UDP-53, port: 53, protocol: UDP}], …}

5. The operator side: derive the ingress overlay, re-sign redis

Now, we do the same for the actual database, and sign-off on the incoming traffic:

no patience? the healed redis profile, as a file

yolo/redis-healed.json is the redis SBOB with the client ingress already merged (it also admits unit 2's web-client). Sign and apply it instead of running overlay.sh:

~/sign-object sign --file yolo/redis-healed.json --key operator.pem --output redis-healed-signed.yaml
kubectl apply -f redis-healed-signed.yaml

The suggested overlay for redis-db is:

kind: ContainerProfile
metadata:
  name: redis-ingress-redis-client
  namespace: redis
spec:
  ingress:
    - identifier: redis-client
      type: internal
      namespaceSelector:
        matchLabels: {kubernetes.io/metadata.name: redis}
      podSelector:
        matchLabels: {app: redis-client}
      ports: [{name: TCP-6379, port: 6379, protocol: TCP}]

The client's suggestion says where it talks to (serviceRefName: redis-master). The vendor shipped redis without knowing this client; allowing it is the operator's decision, signed with the operator key.

~/overlay.sh operator.pem redis "$(kubectl -n redis get $CP -l kubescape.io/profile-role=suggestion -o jsonpath='{.items[0].metadata.name}')"

Output reads:

overlay for redis/redis: +ingress redis-client from redis
[info] Successfully signed object. namespace: redis; name: redis; identity: local-key; issuer: local
2026-09-26T21:14:54Z healed redis/redis (ingress overlay merged, re-signed by the operator)

After the signature is accepted, R0012 stops within about one minute

{"t":"…","level":"info","msg":"Successfully verified object signature"}

6. Quiet — until something new happens

The logs tab goes silent while SET traffic continues. Now do something the profile never saw:

kubectl -n redis exec deploy/redis-client -- id

logs — the exec is judged against YOUR profile now, the shadow is still there and forensically recording

{"rule":"R0001","alert":"Unexpected process launched","ns":"redis","wl":"redis-client","msg":"Unexpected process launched: id with PID 83923"}
{"rule":"R0004","alert":"Linux Capabilities Anomalies in container","ns":"redis","wl":"redis-client","msg":"Unexpected capability used: CAP_SYS_ADMIN in syscall read with PID 82970"}

7. Tamper

sed 's#signature.kubescape.io/signature: ME#signature.kubescape.io/signature: MF#' client-sbob-signed.yaml | kubectl apply -f -

This alert takes a while to appear

{"level":"warn","msg":"Object signature verification failed"}
{"level":"warn","msg":"user-defined ContainerProfile signature mismatch (tamper detected)"}
{
  "rule": "R1016",
  "alert": "Signed profile tampered",
  "ns": "redis",
  "wl": "6cb7c977c8",
  "msg": "Signed ContainerProfile 'redis-client' in namespace 'redis' has been tampered with: 
          signature verification failed: invalid signature: ecdsa: Invalid IEEE_P1363 encoded bytes"
}

Fix it again:

kubectl apply -f client-sbob-signed.yaml

Important

kubectl get $CP lists come back with every spec empty — storage serves metadata only for lists. Take the name from the list, then kubectl get $CP <name> for the object. Signing a list item signs an empty profile: the agent enforces nothing and everything alerts. bobctl validate catches it before you sign.

Fully automated — the system accepts and signs

This part is an aspirational UX, in which we assume that during the suggestion-learn period, the benign traffic is representative. The auto-accept mode should only be used in environments that are free of contamination.

1. Start the auto-acceptor

cd ~/bob/example/redis
pkill -f "[a]cceptor.sh"; NS_SKIP="honey kube-system redis" nohup ~/acceptor.sh operator.pem > acceptor.log 2>&1 &

Only if you skipped unit-1 Deploy a redis-DB with a signed SBOB

Make sure to wait until node-agent's tracers are all active. {"level":"info","msg":"Started tracer"}

cd ~/bob/example/redis
~/sign-object generate-keypair --output operator.pem
export CP=containerprofiles.spdx.softwarecomposition.kubescape.io
kubectl create ns redis
kubectl apply -f vendor/redis-base-signed.yaml
helm install redis oci://registry-1.docker.io/bitnamicharts/redis --version 27.0.18 -n redis --set architecture=standalone --set auth.enabled=false --set image.repository=bitnami/redis --set image.digest=sha256:08863c2c3f4e051fb6139b38fa223e9c13be5033326a59bead182860d899bf98 \
  --set 'master.podLabels.kubescape\.io/user-defined-profile=redis'

Open the split screen and a second tab logs, to watch the node-agent logs

kubectl -n honey logs ds/node-agent -c node-agent --all-pods=true -f | sed -u 's/^\[[^]]*\] //' | jq -Rc --unbuffered 'fromjson? | if .RuleID then {t:.time, rule:.RuleID, alert:.BaseRuntimeMetadata.alertName, ns:.RuntimeK8sDetails.namespace, wl:.RuntimeK8sDetails.workloadName, msg:.message} else {t:.ts, level, msg} end'

2. We add a new client from another namespace

It fetches a web page over TLS and writes a timestamp into redis across namespaces:

export CP=containerprofiles.spdx.softwarecomposition.kubescape.io
kubectl create ns client
kubectl apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-client
  namespace: client
  labels: {app: web-client}
spec:
  replicas: 1
  selector:
    matchLabels: {app: web-client}
  template:
    metadata:
      labels: {app: web-client}
    spec:
      containers:
        - name: client
          image: redis:7-alpine
          command: ["/bin/sh", "-c"]
          args:
            - |
              i=0
              while true; do
                wget -qO /tmp/quote.txt https://labs.iximiuz.com/ || true
                redis-cli -h redis-master.redis.svc.cluster.local -p 6379 SET "quote:$i" "$(date -u +%s)" >/dev/null
                i=$((i+1)); sleep 3
              done
EOF
kubectl -n client rollout status deploy/web-client --timeout=120s

{"t":"…","rule":"R1017","alert":"Rogue artefact","ns":"client","wl":"web-client","msg":"Container 'client' in namespace 'client' has no bound profile — an unmanaged/unknown container is running in the cluster"}
{"t":"…","rule":"R0012","alert":"Unexpected Ingress Network Traffic","ns":"redis","wl":"redis-master","msg":"Unexpected ingress network communication from: 10.42.1.23:6379 using TCP to: redis"}

Note how any new deployment (currently anything that contains a pod and/or job) is marked a ROGUE.

You can heal it if it was your intention. But its shadow collects forensics and the shadow is tamper-proof.

3. Watch it happen (~2–3 min)

no patience? the suggested profile, as a file

yolo/web-client-sbob.json is what the agent suggests for this client. Sign, apply, bind by hand — the acceptor will then leave it alone:

~/sign-object sign --file yolo/web-client-sbob.json --key operator.pem --output web-client-signed.yaml
kubectl apply -f web-client-signed.yaml
kubectl -n client patch deploy web-client -p '{"spec":{"template":{"metadata":{"labels":{"kubescape.io/user-defined-profile":"web-client"}}}}}'

tail -f acceptor.log

2026-09-25T19:23:47Z accepted client/replicaset-web-client-767858fd8c-client-4ae4-9b79-suggested -> Deployment/web-client
overlay for redis/redis: +ingress web-client from client
2026-09-25T19:23:48Z healed redis/redis (ingress overlay merged, re-signed by the operator)

{"t":"…","level":"info","msg":"suggester: SBoBSuggested"}
{"t":"…","level":"info","msg":"container has a user defined profile"}
{"t":"…","level":"info","msg":"Successfully verified object signature"}
{"t":"…","level":"info","msg":"adopted user-authored ContainerProfile as authoritative base"}

discovered vs. portable

Continuously learning shadow profile

Suggestion that is (in this case) auto-accepted 🚨Use with care 🚨

96 execs · 25 opens · 75 syscalls · 6 egress.

spec:
  execs:
    - {path: /bin/busybox, args: [wget, -qO, /tmp/quote.txt, https://labs.iximiuz.com/]}
    - {path: /usr/bin/ssl_client, args: [ssl_client, -s, "3", -n, labs.iximiuz.com]}
    - {path: /bin/busybox, args: [sleep, "3"]}
    - {path: /usr/local/bin/redis-cli, args: [/usr/local/bin/redis-cli, -h, redis-master.redis.svc.cluster.local, -p, "6379", SET, quote:17, "1790417063"]}
    - {path: /usr/local/bin/redis-cli, args: [/usr/local/bin/redis-cli, -h, redis-master.redis.svc.cluster.local, -p, "6379", SET, quote:18, "1790417066"]}
    - … 90 more
  opens:
    - {path: /etc/resolv.conf, flags: [O_CLOEXEC, O_LARGEFILE, O_RDONLY]}
    - {path: /etc/ssl/certs/ca-certificates.crt, flags: [O_RDONLY]}
    - {path: /tmp/quote.txt, flags: [O_CREAT, O_LARGEFILE, O_TRUNC, O_WRONLY]}
    - {path: /tmp/quote.txt, flags: [O_LARGEFILE, O_RDONLY]}
    - … 21 more
  syscalls: [accept4, bind, brk, clone, close, connect, getsockopt, ioctl, poll, … 66 more]
  egress:
    - identifier: 8e352065…
      type: external
      dns: labs.iximiuz.com.
      dnsNames: [labs.iximiuz.com.]
      ipAddress: 104.26.15.92
      ports: [{name: TCP-443, port: 443, protocol: TCP}]
    - identifier: 6eccf30d…
      type: external
      dns: labs.iximiuz.com.
      dnsNames: [labs.iximiuz.com.]
      ipAddress: 104.26.14.92
      ports: [{name: TCP-443, port: 443, protocol: TCP}]
    - identifier: a986953e…
      type: external
      ipAddress: 104.26.15.92
      ports: [{name: TCP-443, port: 443, protocol: TCP}]
    - identifier: db6b8f18…
      type: external
      ipAddress: 104.26.14.92
      ports: [{name: TCP-443, port: 443, protocol: TCP}]
    - … redis-master, kube-dns

11 execs · 25 opens · 4 egress · 1 endpoint · 4 capabilities and syscalls removed

spec:
  architectures: [amd64]
  imageID: docker.io/library/redis@sha256:858f009f…
  matchLabels: {app: web-client}
  capabilities: [CAP_SETGID, CAP_SETPCAP, CAP_SETUID, CAP_SYS_ADMIN]
  execs:
    - {path: /bin/busybox, args: [/bin/sh, ⋯⋯]}
    - {path: /bin/busybox, args: [wget, ⋯⋯]}
    - {path: /bin/busybox, args: [sleep, ⋯⋯]}
    - {path: /usr/bin/ssl_client, args: [ssl_client, ⋯⋯]}
    - {path: /usr/local/bin/redis-cli, args: [/usr/local/bin/redis-cli, -h, redis-master.redis.svc.cluster.local, -p, "6379", SET, ⋯, ⋯]}
    - … 7 more
  opens:
    - {path: /etc/resolv.conf, flags: [O_CLOEXEC, O_LARGEFILE, O_RDONLY]}
    - {path: /etc/ssl/certs/ca-certificates.crt, flags: [O_RDONLY]}
    - {path: /tmp/quote.txt, flags: [O_CREAT, O_LARGEFILE, O_TRUNC, O_WRONLY]}
    - {path: /tmp/quote.txt, flags: [O_LARGEFILE, O_RDONLY]}
    - … 21 more
  egress:
    - identifier: tuned-labs.iximiuz.com
      type: external
      dnsNames: [labs.iximiuz.com]
      ipAddresses: [104.26.14.92]
      ports: [{name: TCP-443, port: 443, protocol: TCP}]
    - identifier: tuned-labs.iximiuz.com-2
      type: external
      dnsNames: [labs.iximiuz.com]
      ipAddresses: [104.26.15.92]
      ports: [{name: TCP-443, port: 443, protocol: TCP}]
    - identifier: 0b4b36aa…
      type: internal
      serviceRefName: redis-master
      serviceRefNamespace: redis
      namespaceSelector:
        matchLabels: {kubernetes.io/metadata.name: redis}
      podSelector:
        matchLabels: {app.kubernetes.io/component: master, app.kubernetes.io/instance: redis, app.kubernetes.io/name: redis}
      ports: [{name: TCP-6379, port: 6379, protocol: TCP}]
    - identifier: f14f2fcf…
      type: internal
      serviceRefName: kube-dns
      serviceRefNamespace: kube-system
      ports: [{name: UDP-53, port: 53, protocol: UDP}]
  endpoints:
    - direction: outbound
      endpoint: ":80/"
      methods: [GET]
      headers: {Host: [labs.iximiuz.com]}

4. Quiet — until something new happens

kubectl -n client exec deploy/web-client -- id

{"rule":"R0001","msg":"Unexpected process launched: id with PID 5064","ns":"client","wl":"web-client"}
{"rule":"R0002","msg":"Unexpected file access detected: id with PID 83923 to /etc/group","ns":"client","wl":"web-client"}
...

5. The agent never overwrites what you signed, only adds

The discovered part, that was added to redis

spec:
  ingress:
    - identifier: redis-client
      type: internal
      namespaceSelector:
        matchLabels: {kubernetes.io/metadata.name: redis}
      podSelector:
        matchLabels: {app: redis-client}
      ports: [{name: TCP-6379, port: 6379, protocol: TCP}]
    - identifier: web-client
      type: internal
      namespaceSelector:
        matchLabels: {kubernetes.io/metadata.name: client}
      podSelector:
        matchLabels: {app: web-client}
      ports: [{name: TCP-6379, port: 6379, protocol: TCP}]

Note

Two keys, two roles: the vendor signed the redis base and is done. Everything added afterwards — the client SBOB and the ingress overlay on redis — is the operator's and carries operator.pem.
In whole-object mode any key verifies, so the roles here are just convention --> see for yourself in the next unit

Fragments — who may sign what

Units 1–2 signed whole objects: one profile, one signature, any key verifies. A bundle splits the profile into fragments, and a root-signed trust policy says per class who may sign and which spec paths the class may touch.

(a) fragments and their signers (b) who may sign what (c) the three checks

The chart ships a policy anchored to a published demo root; its class signers are the two demo keys in keys/.

the trust policy in force (what the agent mounted)

kubectl -n honey get cm node-agent-bundle-policy -o jsonpath='{.data.trust-policy\.json}' | python3 -c 'import sys,json; p=json.load(sys.stdin)["policy"]["classes"]; [print(c, "signers:", [s[:20]+"…" for s in v["signers"]], "paths:", v["allowedSpecPaths"]) for c,v in p.items()]'

base      signers: ['key:162a08d59135652ee0…']  paths: ['architectures', 'capabilities', 'execs', 'opens', 'syscalls', 'seccompProfile', 'endpoints', 'imageID', 'imageTag', 'rulePolicies', 'identifiedCallStacks', 'labelSelector']
overlay   signers: ['key:58fd31c363a8fac06d…']  paths: ['execs', 'opens']
admission signers: ['key:58fd31c363a8fac06d…']  paths: ['ingress', 'egress']

for k in vendor operator; do echo "$k key:$(openssl pkey -in keys/$k.pem -pubout -outform DER 2>/dev/null | sha256sum | cut -c1-64)"; done

vendor   key:162a08d59135652ee095ad2e0d927104c9e4eefe94f4ae844e7387ba1a7352d2
operator key:58fd31c363a8fac06d6eb596c2e8c63d5913b4dc4b971ee62579ec41fff78826

base      signers: ['key:162a08d59135652ee0…']  paths: ['architectures', 'capabilities', 'execs', 'opens', 'syscalls', 'seccompProfile', 'endpoints', 'imageID', 'imageTag', 'rulePolicies', 'identifiedCallStacks', 'labelSelector']
overlay   signers: ['key:58fd31c363a8fac06d…']  paths: ['execs', 'opens']
admission signers: ['key:58fd31c363a8fac06d…']  paths: ['ingress', 'egress']

vendor   key:162a08d59135652ee095ad2e0d927104c9e4eefe94f4ae844e7387ba1a7352d2
operator key:58fd31c363a8fac06d6eb596c2e8c63d5913b4dc4b971ee62579ec41fff78826

1. The vendor ships the base fragment

The SBOB for the redis-db is now labelled as a base fragment of bundle redis, signed with the vendor key:

Same content as the unit-1 base, the commands to resign

export CP=containerprofiles.spdx.softwarecomposition.kubescape.io
cd ~/bob/example/redis && mkdir -p frags
kubectl create --dry-run=client -o json -f vendor/redis-base-signed.yaml | python3 -c '
import sys,json
d=json.load(sys.stdin)
d["metadata"]={"name":"redis-base","namespace":"redis","labels":{"signature.kubescape.io/bundle":"redis","signature.kubescape.io/fragment-class":"base"},"annotations":{"kubescape.io/managed-by":"User"}}
d["spec"]["ingress"]=None
json.dump(d,sys.stdout)' > frags/redis-base.json
~/sign-object sign --file frags/redis-base.json --key keys/vendor.pem --output frags/redis-base-signed.yaml
kubectl apply -f frags/redis-base-signed.yaml
kubectl -n redis delete $CP redis --ignore-not-found

⚠️ Make sure the autoacceptor is killed⚠️

The pod label kubescape.io/user-defined-profile: redis did not change — it now resolves to the bundle redis instead of an object named redis.

[info] Successfully signed object. namespace: redis; name: redis-base; identity: local-key; issuer: local
✓ Profile signed successfully
  Issuer: local
  Identity: local-key
  Timestamp: 1790467128
✓ Signed profile written to: frags/redis-base-signed.yaml
containerprofile.spdx.softwarecomposition.kubescape.io/redis-base created
containerprofile.spdx.softwarecomposition.kubescape.io "redis" deleted from redis namespace

{"level":"info","msg":"assembled signed bundle overlay","bundle":"redis","namespace":"redis","fragments":1,"root":"7c1e…"}
{"rule":"R0012","msg":"Unexpected ingress network communication from: 10.42.0.49:6379 using TCP to: redis","ns":"redis","wl":"redis-master"}

The ingress alerts now stem from the fact, that we only (re)signed the base, not the (network) admission.

2. The operator signs the admission fragment

The ingress both clients need (the redis-client from unit 1 and the web-client from unit 2) as one admission fragment, signed with the operator key:

Same content as the unit-1 overlay, the commands to resign

cat > frags/redis-client-ingress.json <<'EOF'
{"apiVersion":"spdx.softwarecomposition.kubescape.io/v1beta1","kind":"ContainerProfile",
 "metadata":{"name":"redis-client-ingress","namespace":"redis","labels":{"signature.kubescape.io/bundle":"redis","signature.kubescape.io/fragment-class":"admission"},"annotations":{"kubescape.io/managed-by":"User"}},
 "spec":{"ingress":[{"identifier":"redis-client","type":"internal",
   "namespaceSelector":{"matchLabels":{"kubernetes.io/metadata.name":"redis"}},
   "podSelector":{"matchLabels":{"app":"redis-client"}},
   "ports":[{"name":"TCP-6379","port":6379,"protocol":"TCP"}]},
   {"identifier":"web-client","type":"internal",
   "namespaceSelector":{"matchLabels":{"kubernetes.io/metadata.name":"client"}},
   "podSelector":{"matchLabels":{"app":"web-client"}},
   "ports":[{"name":"TCP-6379","port":6379,"protocol":"TCP"}]}]}}
EOF
~/sign-object sign --file frags/redis-client-ingress.json --key keys/operator.pem --output frags/redis-client-ingress-signed.yaml
kubectl apply -f frags/redis-client-ingress-signed.yaml

[info] Successfully signed object. namespace: redis; name: redis-client-ingress; identity: local-key; issuer: local
✓ Profile signed successfully
  Issuer: local
  Identity: local-key
  Timestamp: 1790467305
✓ Signed profile written to: frags/redis-client-ingress-signed.yaml
containerprofile.spdx.softwarecomposition.kubescape.io/redis-client-ingress created

kind: ContainerProfile
metadata:
  name: redis-client-ingress
  namespace: redis
  labels:
    signature.kubescape.io/bundle: redis
    signature.kubescape.io/fragment-class: admission
  annotations:
    kubescape.io/managed-by: User
    signature.kubescape.io/identity: local-key
    signature.kubescape.io/signature: MEQCIB…
    signature.kubescape.io/content: H4sIAAAA…
spec:
  ingress:
    - identifier: redis-client
      type: internal
      namespaceSelector:
        matchLabels: {kubernetes.io/metadata.name: redis}
      podSelector:
        matchLabels: {app: redis-client}
      ports: [{name: TCP-6379, port: 6379, protocol: TCP}]
    - identifier: web-client
      type: internal
      namespaceSelector:
        matchLabels: {kubernetes.io/metadata.name: client}
      podSelector:
        matchLabels: {app: web-client}
      ports: [{name: TCP-6379, port: 6379, protocol: TCP}]

"msg":"assembled signed bundle overlay","bundle":"redis","namespace":"redis","fragments":2,"root":"a94f…"

Now, the Unexpected ingress alerts stop for both clients. If you skipped unit 2 there is no web-client; the second entry then admits nothing and is harmless.

3. The operator signs off another overlay to allow `df` execution

cat > frags/redis-ops-overlay.json <<'EOF'
{"apiVersion":"spdx.softwarecomposition.kubescape.io/v1beta1","kind":"ContainerProfile",
 "metadata":{"name":"redis-ops-overlay","namespace":"redis","labels":{"signature.kubescape.io/bundle":"redis","signature.kubescape.io/fragment-class":"overlay"},"annotations":{"kubescape.io/managed-by":"User"}},
 "spec":{"execs":[{"path":"/usr/bin/df","args":["/usr/bin/df","-h"]}]}}
EOF
~/sign-object sign --file frags/redis-ops-overlay.json --key keys/operator.pem --output frags/redis-ops-overlay-signed.yaml
kubectl apply -f frags/redis-ops-overlay-signed.yaml
sleep 60
kubectl -n redis exec sts/redis-master -- df -h >/dev/null
kubectl -n redis exec sts/redis-master -- id >/dev/null

df -h is silent, id is not

"msg":"assembled signed bundle overlay","bundle":"redis","namespace":"redis","fragments":3,"root":"e21b…"

{"rule":"R0001","msg":"Unexpected process launched: id with PID …","ns":"redis","wl":"redis-master"}

4. Three ways to be refused

4.1) Wrong `key` for the class

~/sign-object sign --file frags/redis-client-ingress.json --key keys/vendor.pem --output frags/wrongkey.yaml
kubectl apply -f frags/wrongkey.yaml

[info] Successfully signed object. namespace: redis; name: redis-client-ingress; identity: local-key; issuer: local
✓ Profile signed successfully
  Issuer: local
  Identity: local-key
  Timestamp: 1790469773
✓ Signed profile written to: frags/wrongkey.yaml

{"level":"warn","msg":"signed bundle overlay refused: a verified member no longer verifies; keeping the last verified composite"}
{"level":"warn","msg":"refreshOneEntry: bundle re-assembly refused; keeping the last verified profile"}

Put the operator-signed admission fragment back before the next attempt, else the bundle stays refused for the wrong reason:

kubectl apply -f frags/redis-client-ingress-signed.yaml

4.2) Wrong `paths` for the class

python3 -c '
import json
d=json.load(open("frags/redis-client-ingress.json")); d["metadata"]["name"]="redis-sneaky"
d["spec"]["execs"]=[{"path":"/bin/bash","args":["/bin/bash"]}]
json.dump(d,open("frags/sneaky.json","w"))'
~/sign-object sign --file frags/sneaky.json --key keys/operator.pem --output frags/sneaky-signed.yaml
kubectl apply -f frags/sneaky-signed.yaml

[info] Successfully signed object. namespace: redis; name: redis-sneaky; identity: local-key; issuer: local
✓ Profile signed successfully
  Issuer: local
  Identity: local-key
  Timestamp: 1790469996
✓ Signed profile written to: frags/sneaky-signed.yaml

{
  "level": "warn",
  "msg": "signed bundle overlay: dropped non-member object(s)
          labelled into the bundle by an unauthorised writer"
}
{"level":"warn","msg":"refreshOneEntry: bundle re-assembly refused; keeping the last verified profile"}

4.3) Tampered content

kubectl -n redis get $CP redis-ops-overlay -o jsonpath='{.metadata.annotations.signature\.kubescape\.io/content}' \
  | python3 -c 'import sys,base64,gzip; d=gzip.decompress(base64.b64decode(sys.stdin.read())); print(base64.b64encode(gzip.compress(d.replace(b"/usr/bin/df", b"/bin/backdoor"))).decode())' \
  | xargs -I{} kubectl -n redis annotate $CP redis-ops-overlay --overwrite signature.kubescape.io/content={}

{
  "level": "warn",
  "msg": "signed fragment stored spec diverges from the signed content: the stored spec is display-only and is NOT enforced; enforcement uses the signed content"
}
{"level":"warn","msg":"Object signature verification failed"}
{
  "rule": "R1016",
  "ns": "redis",
  "wl": null,
  "msg": "Signed ContainerProfile bundle 'redis' in namespace 'redis' has been tampered with:   
          fragment signature does not verify (tampered): \"redis-ops-overlay\": signature verification failed: 
          invalid signature: invalid signature when validating ASN.1 encoded signature"
}

5. Leaves stay intact in the bundle

for f in redis-base redis-client-ingress redis-ops-overlay; do
  kubectl -n redis get $CP $f -o yaml > /tmp/leaf.yaml && ~/sign-object verify --file /tmp/leaf.yaml --strict=false >/dev/null && echo "leaf $f: OK"
done

[info] Successfully verified object signature. namespace: redis; name: redis-base; identity: local-key; issuer: local
leaf redis-base: OK
[info] Successfully verified object signature. namespace: redis; name: redis-client-ingress; identity: local-key; issuer: local
leaf redis-client-ingress: OK
[warning] Object signature verification failed. namespace: redis; name: redis-ops-overlay; error: invalid signature: invalid signature when validating ASN.1 encoded signature
Error: signature verification failed: signature verification failed: invalid signature: invalid signature when validating ASN.1 encoded signature

Clean up the three and re-ship the good overlay:

kubectl -n redis delete $CP redis-sneaky
kubectl apply -f frags/redis-client-ingress-signed.yaml
kubectl apply -f frags/redis-ops-overlay-signed.yaml

Now, you should get an OK for all of them.

Important

The keys in keys/ and the root the chart trusts are published demo material — the agent says so at boot (anchored to the PUBLISHED DEMO root key: this authenticates nothing). A real deployment mounts its own root public key, ships its own policy, and keeps every private key off the cluster.

Previous lesson: Software Bill of Behaviour for CNCF projects

Course progress

The agent suggests the client's SBOB — you sign it

Suggest, prove, sign, bind

0. One terminal for logs

1. The vendor ships redis with a signed SBOB — no external dependecies

2. A rogue client appears

3. The agent suggests a profile (~2 min)

4. You sign it with your key and bind it

5. The operator side: derive the ingress overlay, re-sign redis

6. Quiet — until something new happens

7. Tamper

Fully automated — the system accepts and signs

1. Start the auto-acceptor

2. We add a new client from another namespace

3. Watch it happen (~2–3 min)

4. Quiet — until something new happens

5. The agent never overwrites what you signed, only adds

Fragments — who may sign what

1. The vendor ships the base fragment

2. The operator signs the admission fragment

3. The operator signs off another overlay to allow df execution

4. Three ways to be refused

4.1) Wrong key for the class

4.2) Wrong paths for the class

4.3) Tampered content

5. Leaves stay intact in the bundle

3. The operator signs off another overlay to allow `df` execution

4.1) Wrong `key` for the class

4.2) Wrong `paths` for the class