This lesson was created by a community author. Community content is reviewed by the iximiuz Labs team on a best effort basis.

Lesson in Software Bill of Behavior - vendor supplied runtime profile for tampering and anomaly detection

The agent suggests the client's SBOB — you sign it

A vendor-signed redis, an unprotected client, and node-agent proposing the client's portable profile on its own. Prove it equals bobctl, sign it, bind it, watch the alerts stop.

Propose, author, sign, heal

0. One terminal for logs

Open a second tab logs, run this once and leave it:

kubectl -n honey logs ds/node-agent -c node-agent --all-pods=true -f | sed -u 's/^\[[^]]*\] //' | jq -Rc --unbuffered 'fromjson? | if .RuleID then {t:.time, rule:.RuleID, alert:.BaseRuntimeMetadata.alertName, ns:.RuntimeK8sDetails.namespace, wl:.RuntimeK8sDetails.workloadName, msg:.message} else {t:.ts, level, msg} end'

ready when you see this — the suggester line and every tracer on every node

{"t":"…","level":"info","msg":"suggester enabled"}
{"t":"…","level":"info","msg":"Started tracer"}
{"t":"…","level":"info","msg":"ContainerWatcher started successfully"}
{"t":".","level":"info","msg":"RogueArtifact writer wired: discoverable signoff state enabled"}
{"t":".","level":"info","msg":"queue processor started"}
{"t":".","level":"info","msg":"Successfully verified object signature"}
{"t":".","level":"info","msg":"signed bundle overlays enabled in alert mode"}
{"t":".","level":"warn","msg":"signed bundle overlays anchored to the PUBLISHED DEMO root key: this authenticates nothing, mount a real root before relying on signatures"}
{"t":".","level":"warn","msg":"rule signing DISABLED: the trust policy in force carries no ruleClasses; ANY Rules object in ANY namespace will load without a signature check"}
{"t":".","level":"info","msg":"trust policy in force"}

1. The vendor ships redis with a signed SBOB — no client is known yet

_redis with SBOB signed by vendor _

cd ~/bob/example/redis
export CP=containerprofiles.spdx.softwarecomposition.kubescape.io
kubectl create ns redis
kubectl apply -f vendor/redis-base-signed.yaml
helm install redis oci://registry-1.docker.io/bitnamicharts/redis --version 27.0.18 -n redis --set architecture=standalone --set auth.enabled=false --set image.repository=bitnami/redis --set image.digest=sha256:08863c2c3f4e051fb6139b38fa223e9c13be5033326a59bead182860d899bf98 \
  --set 'master.podLabels.kubescape\.io/user-defined-profile=redis'

logs

{"t":".","level":"warn","msg":"signed fragment stored spec diverges from the signed content: the stored spec is display-only and is NOT enforced; enforcement uses the signed content"}
{"t":".","level":"info","msg":"adopted user-authored ContainerProfile as authoritative base"}
{"t":"…","level":"info","msg":"container has a user defined profile"}
{"t":"…","level":"info","msg":"Successfully verified object signature"}

The reason for the tamper alert is that during the first tick, the operator hasnt reconciled

the vendor's SBOB, abridged — 15 execs · 79 opens · 8 capabilities · no ingress

kind: ContainerProfile
metadata:
  name: redis
  namespace: redis
  annotations:
    kubescape.io/managed-by: User
    signature.kubescape.io/identity: local-key
    signature.kubescape.io/issuer: local
    signature.kubescape.io/timestamp: "1790366987"
    signature.kubescape.io/signature: MEUCIQDd5Ba9Hg2iSh2OAO8w…
    signature.kubescape.io/certificate: LS0tLS1CRUdJTiBDRVJUSUZJ…
    signature.kubescape.io/content: H4sIAAAAAAAA/7xZ73LiOBJ/…
spec:
  matchLabels: {app.kubernetes.io/component: master, app.kubernetes.io/instance: redis, app.kubernetes.io/name: redis}
  capabilities: [CAP_CHOWN, CAP_DAC_OVERRIDE, CAP_NET_ADMIN, CAP_SETGID, CAP_SETUID, CAP_SYS_ADMIN, …]
  execs:
    - {args: [/opt/bitnami/redis/bin/redis-cli, -h, localhost, -p, "6379", ping], path: /opt/bitnami/redis/bin/redis-cli}
    - {args: [/opt/bitnami/redis/bin/redis-server, ⋯⋯], path: /opt/bitnami/redis/bin/redis-server}
    - {args: [/health/ping_liveness_local.sh, ⋯⋯], path: /usr/bin/bash}
    - …
  opens:
    - {flags: [O_RDONLY], path: /data}
    - {flags: [O_APPEND, O_CREAT, O_WRONLY], path: /data/appendonlydir/appendonly.aof.1.incr.aof}
    - …
  ingress: null

2. A client appears — unprotected

no profile, no trust: the client is a rogue artefact

sed '/kubescape.io\/user-defined-profile/d' client.yaml | kubectl apply -f -
kubectl -n redis rollout status deploy/redis-client --timeout=120s

Alerts — R1017 Rogue artefact; R0012 Unexpected Ingress Network Traffic on redis for every connection; and, for the first two minutes, everything the client does

{"t":"…","rule":"R1017","alert":"Rogue artefact","ns":"redis","wl":"redis-client","msg":"Container 'client' in namespace 'redis' has no bound profile — an unmanaged/unknown container is running in the cluster"}
{"t":"…","rule":"R0001","alert":"Unexpected process launched","ns":"redis","wl":"redis-client","msg":"Unexpected process launched: redis-cli with PID 76287"}
{"t":"…","rule":"R0011","alert":"Unexpected Egress Network Traffic","ns":"redis","wl":"redis-client","msg":"Unexpected egress network communication to: 10.43.x.x:6379 using TCP from: client"}
{"t":"…","rule":"R0040","alert":"Unexpected process arguments","ns":"redis","wl":"redis-client","msg":"Unexpected process arguments: redis-cli with PID 76301"}
{"t":"…","rule":"R0012","alert":"Unexpected Ingress Network Traffic","ns":"redis","wl":"redis-master","msg":"Unexpected ingress network communication from: 10.42.0.49:6379 using TCP to: redis"}

An unbound container is deny-all from its first event: every distinct behaviour alerts once, and every SET k:$i is a distinct behaviour, so R0040 fires on each loop. That is what an unknown container in production looks like. After two minutes its own shadow judges it and the client's own loop goes quiet. R0012 on redis stays, because nobody admitted the client there.

3. You ask, the agent proposes the admission (~2 min)

The agent proposes nothing on its own. When a shadow has learned for the two-minute window it says so, once, on the pod:

kubectl -n redis get events --field-selector reason=SBoBSuggestionAvailable -o jsonpath='{range .items[*]}{.involvedObject.name}{"\t"}{.message}{"\n"}{end}'

redis-client-…   Shadow replicaset-redis-client-…-client-… is ready. Request a suggestion with: bobctl suggest -n redis redis-client-…

A request is one annotation on one live pod; no rollout. Ask both sides: the client, which says where it sends, and redis, which says what it received:

kubectl -n redis annotate pod -l app=redis-client --overwrite kubescape.io/suggest-request=$(date +%s)-$(openssl rand -hex 3)
kubectl -n redis annotate pod redis-master-0 --overwrite kubescape.io/suggest-request=$(date +%s)-$(openssl rand -hex 3)
watch -n5 "kubectl -n redis get $CP -l kubescape.io/delta-of=redis -o custom-columns=NAME:.metadata.name,EVIDENCE:.metadata.annotations.kubescape\.io/evidence,REQUEST:.metadata.annotations.kubescape\.io/suggest-request"

NAME                            EVIDENCE   REQUEST
redis-peer-redis-redis-client   both       1790…-a1b2c3

Every request ends in exactly one terminal Event on the pod, its note ending in request=<id>: SBoBDeltaProposed when a delta was written, SBoBSuggestionEmpty when nothing is missing from the pod's own profile, SBoBSuggestionRejected or SBoBSuggestionFailed otherwise. A proposal written on another workload is announced on top of that, as SBoBPeerProposed. The same id asked twice is a no-op; a new id runs again.

kubectl -n redis get events --field-selector reason=SBoBSuggestionEmpty -o jsonpath='{range .items[*]}{.involvedObject.name}{"\t"}{.message}{"\n"}{end}'

redis-client-…   Nothing new to propose for redis-client-…: the behaviour is already covered (network-scope). request=1791023688-1d10ec
redis-master-0   Nothing new to propose for redis-master-0: the behaviour is already covered (delta-empty). request=1791023689-3f625f

The client's own answer is network-scope: it is unbound, and what it executes and opens is not the agent's to propose. Its request still produced the proposal on redis.

kubectl -n redis get events --field-selector reason=SBoBPeerProposed -o jsonpath='{.items[0].message}{"\n"}'

the proposal — the agent's ingress entry for redis, not yet signed

kind: ContainerProfile
metadata:
  name: redis-peer-redis-redis-client
  namespace: redis
  labels: {kubescape.io/profile-role: suggestion, kubescape.io/delta-of: redis, kubescape.io/peer-workload: redis-client}
  annotations: {kubescape.io/evidence: both, kubescape.io/created-by: node-agent-suggester, kubescape.io/suggest-request: 1790…-a1b2c3, kubescape.io/expires-at: …}
spec:
  matchLabels: {app.kubernetes.io/component: master, app.kubernetes.io/instance: redis, app.kubernetes.io/name: redis}
  ingress:
    - identifier: redis-redis-client
      type: internal
      namespaceSelector:
        matchLabels: {kubernetes.io/metadata.name: redis}
      podSelector:
        matchLabels: {app: redis-client}
      ports: [{name: TCP-6379, port: 6379, protocol: TCP}]

client = the client's request found it in the client's generalised egress. shadow = redis's request found it in redis's shadow ingress. both = the two agree. The agent proposes network only: who talks to whom, on which port. What a container executes and opens is yours to author, next step.

4. You author the client's SBOB from what the agent learned, sign it, bind it

The agent's learning for the client is a shadow profile: everything it saw, literally. bobctl turns it into something portable, exactly the way the agent does for its own proposals:

~/sign-object generate-keypair --output operator.pem
N=$(kubectl -n redis get $CP -l 'kubescape.io/workload-name=redis-client,kubescape.io/profile-role!=suggestion' -o jsonpath='{.items[0].metadata.name}')
mkdir -p shadow && kubectl -n redis get $CP "$N" -o yaml > shadow/client.yaml
~/bobctl generalize -d shadow --sbob redis-client --namespace redis -o client-portable.yaml
~/bobctl portable --file client-portable.yaml --out client-sbob.yaml
~/bobctl validate --file client-sbob.yaml
~/sign-object sign --file client-sbob.yaml --key operator.pem --output client-sbob-signed.yaml
kubectl apply -f client-sbob-signed.yaml
kubectl -n redis patch deploy redis-client -p '{"spec":{"template":{"metadata":{"labels":{"kubescape.io/user-defined-profile":"redis-client"}}}}}'
kubectl -n redis rollout status deploy/redis-client --timeout=120s

no patience? the portable profile, as a file

yolo/client-sbob.json is what the pipeline above produces for this client:

~/sign-object sign --file yolo/client-sbob.json --key operator.pem --output client-sbob-signed.yaml && kubectl apply -f client-sbob-signed.yaml
kubectl -n redis patch deploy redis-client -p '{"spec":{"template":{"metadata":{"labels":{"kubescape.io/user-defined-profile":"redis-client"}}}}}'

learned vs. portable

shadow — what was learned

portable — what you sign

125 execs · 13 opens · 63 syscalls · 2 egress

spec:
  execs:
    - {path: /usr/local/bin/redis-cli, args: [/usr/local/bin/redis-cli, -h, redis-master, -p, "6379", SET, k:32, v:32]}
    - {path: /usr/local/bin/redis-cli, args: [/usr/local/bin/redis-cli, -h, redis-master, -p, "6379", SET, k:33, v:33]}
    - … 119 more redis-cli lines, one per SET
  egress:
    - identifier: f8ed3623…
      type: internal
      serviceRefName: redis-master
      serviceRefNamespace: redis
      podSelector:
        matchLabels: {app.kubernetes.io/component: master, app.kubernetes.io/instance: redis, app.kubernetes.io/name: redis}
      ports: [{name: TCP-6379, port: 6379, protocol: TCP}]

5. The operator signs the admission into redis

The vendor shipped redis without knowing this client; admitting it is the operator's decision. Merge the proposal into the vendor SBOB and re-sign with the operator key:

kubectl -n redis get $CP redis -o json | jq --argjson add "$(kubectl -n redis get $CP redis-peer-redis-redis-client -o json | jq '.spec.ingress')" '.spec.ingress = ((.spec.ingress // []) + $add) | {apiVersion, kind, metadata: {name: .metadata.name, namespace: .metadata.namespace, annotations: {"kubescape.io/managed-by": "User"}}, spec}' > redis-healed.json
~/sign-object sign --file redis-healed.json --key operator.pem --output redis-healed-signed.yaml && kubectl apply -f redis-healed-signed.yaml

no patience? the healed redis, as a file

yolo/redis-healed.json is the redis SBOB with the client ingress already merged (it also admits unit 2's web-client):

~/sign-object sign --file yolo/redis-healed.json --key operator.pem --output redis-healed-signed.yaml && kubectl apply -f redis-healed-signed.yaml

logs — redis re-verified, R0012 stops

{"t":"…","level":"info","msg":"Successfully verified object signature"}
{"t":"…","level":"info","msg":"adopted user-authored ContainerProfile as authoritative base"}

The proposal is still there: the agent withdraws what it wrote only when it is asked again and finds the entry admitted. Ask redis once more:

kubectl -n redis annotate pod redis-master-0 --overwrite kubescape.io/suggest-request=$(date +%s)-$(openssl rand -hex 3)

{"t":"…","level":"info","msg":"suggester: peer proposal admitted by the governing profile, withdrawn","namespace":"redis","name":"redis-peer-redis-redis-client"}
{"t":"…","level":"info","msg":"suggester: peer proposal skipped","slug":"statefulset-redis-master-…","egress":"tuned-10.42.1.32","target":"","reason":"NotAPodPeer"}

and one proposal you should not sign — the shadow remembers a pod that is gone

The same answer writes redis-delta-admission: redis's shadow still holds the address of the first client pod, the unbound one that was replaced when you bound the deployment. Nothing resolves that address to a pod any more, so the generaliser can only call it external:

spec:
  ingress:
    - identifier: tuned-10.42.1.32
      type: external
      ipAddresses: [10.42.1.32]
      ports: [{name: TCP-6379, port: 6379, protocol: TCP}]

A pod address is not an identity. Leave it; it expires after a day. This is the shape every stale address takes, and the reason the runner in unit 2 §8 proves itself across a full restart.

6. Detection, then the correction

Make the client talk to something its profile does not name. A port on redis nobody opened:

kubectl -n redis exec deploy/redis-client -- redis-cli -h redis-master -p 6380 ping

logs — judged against your profile

{"t":"…","rule":"R0011","alert":"Unexpected Egress Network Traffic","ns":"redis","wl":"redis-client","msg":"Unexpected egress network communication to: 10.43.x.x:6380 using TCP from: client"}

Ask the client. It is bound now, so the answer is a delta: only what its signed profile does not grant.

kubectl -n redis annotate pod -l app=redis-client --overwrite kubescape.io/suggest-request=$(date +%s)-$(openssl rand -hex 3)
kubectl -n redis get $CP -l kubescape.io/delta-of=redis-client -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.annotations.kubescape\.io/suggestion-version}{"\n"}{end}'

redis-client-delta-admission   1

The same request refreshes the other side of the link: redis does not admit 6380 from this client either, so redis-peer-redis-redis-client is back, now naming both ports.

the delta — only the network the signed profile does not grant

spec:
  matchLabels: {app: redis-client}
  egress:
    - type: internal
      podSelector:
        matchLabels: {app.kubernetes.io/component: master, app.kubernetes.io/instance: redis, app.kubernetes.io/name: redis}
      ports: [{name: TCP-6380, port: 6380, protocol: TCP}]

That is the whole proposal. Not the command that caused it, not the files it touched: the agent proposes network, and leaves the rest to the author.

If 6380 belongs in the profile, merge and re-sign, same move as step 5. If it does not, leave it: the alert was the point, and a proposal nobody signs expires after a day. The agent never touches what you signed.

7. Tamper

sed 's#signature.kubescape.io/signature: ME#signature.kubescape.io/signature: MF#' client-sbob-signed.yaml | kubectl apply -f -

logs (within a minute)

{"t":"…","level":"warn","msg":"Object signature verification failed"}
{"t":"…","level":"warn","msg":"user-defined ContainerProfile signature mismatch (tamper detected)"}

kubectl apply -f client-sbob-signed.yaml

Important

kubectl get $CP lists come back with every spec empty — storage serves metadata only for lists. Take the name from the list, then kubectl get $CP <name> for the object. Signing a list item signs an empty profile: the agent enforces nothing and everything alerts. bobctl validate catches it before you sign.

Autodiscovered — the agent proposes, the operator signs

starting here? unit 2 needs redis bound to the vendor SBOB and your operator key, nothing else from unit 1

cd ~/bob/example/redis
export CP=containerprofiles.spdx.softwarecomposition.kubescape.io
kubectl create ns redis
kubectl apply -f vendor/redis-base-signed.yaml
helm install redis oci://registry-1.docker.io/bitnamicharts/redis --version 27.0.18 -n redis --set architecture=standalone --set auth.enabled=false --set image.repository=bitnami/redis --set image.digest=sha256:08863c2c3f4e051fb6139b38fa223e9c13be5033326a59bead182860d899bf98 \
  --set 'master.podLabels.kubescape\.io/user-defined-profile=redis'
~/sign-object generate-keypair --output operator.pem

1. A different client, in its own namespace

It fetches a web page over TLS and writes a timestamp into redis across namespaces:

cd ~/bob/example/redis
export CP=containerprofiles.spdx.softwarecomposition.kubescape.io
kubectl create ns client
kubectl apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-client
  namespace: client
  labels: {app: web-client}
spec:
  replicas: 1
  selector:
    matchLabels: {app: web-client}
  template:
    metadata:
      labels: {app: web-client}
    spec:
      containers:
        - name: client
          image: redis:7-alpine
          command: ["/bin/sh", "-c"]
          args:
            - |
              i=0
              while true; do
                wget -qO /tmp/quote.txt https://labs.iximiuz.com/ || true
                redis-cli -h redis-master.redis.svc.cluster.local -p 6379 SET "quote:$i" "$(date -u +%s)" >/dev/null
                i=$((i+1)); sleep 3
              done
EOF
kubectl -n client rollout status deploy/web-client --timeout=120s

logs — R1017 on the client; R0012 on redis, because nobody allowed this client

{"t":"…","rule":"R1017","alert":"Rogue artefact","ns":"client","wl":"web-client","msg":"Container 'client' in namespace 'client' has no bound profile — an unmanaged/unknown container is running in the cluster"}
{"t":"…","rule":"R0012","alert":"Unexpected Ingress Network Traffic","ns":"redis","wl":"redis-master","msg":"Unexpected ingress network communication from: 10.42.1.23:6379 using TCP to: redis"}

2. Ask, and one proposal appears, across namespaces (~2–3 min)

Two minutes after the client started, its shadow is announced (SBoBSuggestionAvailable on the pod). Ask the client, then ask redis:

kubectl -n client annotate pod -l app=web-client --overwrite kubescape.io/suggest-request=$(date +%s)-$(openssl rand -hex 3)
kubectl -n redis annotate pod redis-master-0 --overwrite kubescape.io/suggest-request=$(date +%s)-$(openssl rand -hex 3)
kubectl get $CP -A -l kubescape.io/delta-of -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{"\t"}{.metadata.labels.kubescape\.io/delta-of}{"\t"}{.metadata.annotations.kubescape\.io/evidence}{"\n"}{end}'

redis/redis-peer-client-web-client     redis   both

Nobody wrote a script. The client's request found redis in the client's generalised egress, redis's request found the client in redis's shadow ingress, and the proposal landed on redis, across namespaces, because the installer set suggester.peers.crossNamespace=true. The client itself gets no proposal: it is unbound, so nothing governs it yet, and what it executes is not the agent's to suggest; its own request ends in SBoBSuggestionEmpty.

kubectl -n client get events --field-selector reason=SBoBPeerProposed -o jsonpath='{.items[0].message}{"\n"}'

redis/redis-master receives traffic from client/web-client that its profile redis does not admit. Proposal redis-peer-client-web-client (evidence: both). Review, sign and apply it as an admission fragment of redis, or merge it into redis.

3. The operator authors the client, signs it, binds it

yolo/web-client-sbob.json is the portable profile for this client, with the SET quote:$i <timestamp> arguments collapsed to SET, ⋯, ⋯. Sign it, apply it, bind:

~/sign-object sign --file yolo/web-client-sbob.json --key operator.pem --output web-client-signed.yaml && kubectl apply -f web-client-signed.yaml
kubectl -n client patch deploy web-client -p '{"spec":{"template":{"metadata":{"labels":{"kubescape.io/user-defined-profile":"web-client"}}}}}'
kubectl -n client rollout status deploy/web-client --timeout=120s

authoring it yourself from the shadow, as in unit 1 §4 — and why the file is the default here

N=$(kubectl -n client get $CP -l 'kubescape.io/workload-name=web-client,kubescape.io/profile-role!=suggestion' -o jsonpath='{.items[0].metadata.name}')
mkdir -p shadow2 && kubectl -n client get $CP "$N" -o yaml > shadow2/web-client.yaml
~/bobctl generalize -d shadow2 --sbob web-client --namespace client -o web-client-portable.yaml
~/bobctl portable --file web-client-portable.yaml --out web-client-sbob.yaml
~/bobctl validate --file web-client-sbob.yaml

With bobctl v0.1.6 this comes back valid with 61 execs, 37 of them one redis-cli … SET quote:N <timestamp> line each: the value changes on every call and the collapse does not fold it. Sign that and R0040 "Unexpected process arguments" keeps firing on every SET, correctly, because the profile names arguments that never recur. Collapse those lines to SET, ⋯, ⋯ by hand (the yolo file is exactly that) before you sign. The agent's own proposals are not affected: they are network only.

learned vs. portable — this client

The shadow learns one exec per call (SET quote:17 1790417063, SET quote:18 …), addresses one per resolution; the portable profile you sign names 11 execs, 0 syscalls and 4 egress by identity:

spec:
  matchLabels: {app: web-client}
  execs:
    - {path: /bin/busybox, args: [wget, ⋯⋯]}
    - {path: /usr/bin/ssl_client, args: [ssl_client, ⋯⋯]}
    - {path: /usr/local/bin/redis-cli, args: [/usr/local/bin/redis-cli, -h, redis-master.redis.svc.cluster.local, -p, "6379", SET, ⋯, ⋯]}
    - …
  egress:
    - identifier: tuned-labs.iximiuz.com
      type: external
      dnsNames: [labs.iximiuz.com]
      ports: [{name: TCP-443, port: 443, protocol: TCP}]
    - type: internal
      namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: redis}}
      podSelector: {matchLabels: {app.kubernetes.io/component: master, app.kubernetes.io/instance: redis, app.kubernetes.io/name: redis}}
      ports: [{name: TCP-6379, port: 6379, protocol: TCP}]
    - type: internal
      namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: kube-system}}
      podSelector: {matchLabels: {k8s-app: kube-dns}}
      ports: [{name: UDP-53, port: 53, protocol: UDP}]

4. The operator admits the client into redis

Same move as unit 1, with the entry the agent proposed:

kubectl -n redis get $CP redis -o json | jq --argjson add "$(kubectl -n redis get $CP redis-peer-client-web-client -o json | jq '.spec.ingress')" '.spec.ingress = ((.spec.ingress // []) + $add) | {apiVersion, kind, metadata: {name: .metadata.name, namespace: .metadata.namespace, annotations: {"kubescape.io/managed-by": "User"}}, spec}' > redis-healed.json
~/sign-object sign --file redis-healed.json --key operator.pem --output redis-healed-signed.yaml && kubectl apply -f redis-healed-signed.yaml

what redis admits now — two clients, two namespaces

spec:
  ingress:
    - identifier: redis-redis-client
      type: internal
      namespaceSelector:
        matchLabels: {kubernetes.io/metadata.name: redis}
      podSelector:
        matchLabels: {app: redis-client}
      ports: [{name: TCP-6379, port: 6379, protocol: TCP}]
    - identifier: client-web-client
      type: internal
      namespaceSelector:
        matchLabels: {kubernetes.io/metadata.name: client}
      podSelector:
        matchLabels: {app: web-client}
      ports: [{name: TCP-6379, port: 6379, protocol: TCP}]

logs — verified, adopted, R0012 stops

{"t":"…","level":"info","msg":"Successfully verified object signature"}
{"t":"…","level":"info","msg":"adopted user-authored ContainerProfile as authoritative base"}

The proposal stays until redis is asked again; then the agent finds it admitted and withdraws it:

kubectl -n redis annotate pod redis-master-0 --overwrite kubescape.io/suggest-request=$(date +%s)-$(openssl rand -hex 3)

{"t":"…","level":"info","msg":"suggester: peer proposal admitted by the governing profile, withdrawn","namespace":"redis","name":"redis-peer-client-web-client"}

5. Quiet — until something new happens on the network, and then a proposal

kubectl -n client exec deploy/web-client -- wget -qO /dev/null https://example.com/

logs

{"t":"…","rule":"R0011","alert":"Unexpected Egress Network Traffic","ns":"client","wl":"web-client","msg":"Unexpected egress network communication to: 93.184.215.14:443 using TCP from: client"}
{"t":"…","rule":"R0005","alert":"Unexpected domain request","ns":"client","wl":"web-client","msg":"Unexpected domain communication: example.com. from: client"}

The alert is the detection. Give the shadow one update period (30 s) to record the flow, then ask the client, and the agent adds the correction, as a proposal you can sign or ignore. Ask too early and the honest answer is SBoBSuggestionEmpty; ask again:

sleep 30
kubectl -n client annotate pod -l app=web-client --overwrite kubescape.io/suggest-request=$(date +%s)-$(openssl rand -hex 3)
kubectl -n client get $CP -l kubescape.io/delta-of=web-client -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.annotations.kubescape\.io/suggestion-version}{"\n"}{end}'

web-client-delta-admission    1

the delta — only the network the signed profile does not grant

spec:
  matchLabels: {app: web-client}
  egress:
    - type: external
      dnsNames: [example.com]
      ports: [{name: TCP-443, port: 443, protocol: TCP}]

Merge and re-sign as in step 4 if example.com belongs in the profile. Leave it and it expires after a day. Ask again with a new id and the same behaviour: SBoBSuggestionEmpty, the proposal keeps its version. Nothing runs between your requests.

Now run something the profile never saw: kubectl -n client exec deploy/web-client -- id, then ask again. R0001 fires, and no proposal follows. Processes and files are the author's; the agent proposes network.

6. The agent never overwrites what you signed

sleep 120
kubectl get $CP -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{"\t"}{.metadata.annotations.signature\.kubescape\.io/identity}{"\n"}{end}' | grep -v $'\t$'

client/web-client     local-key
redis/redis           local-key
redis/redis-client    local-key

Proposals carry kubescape.io/created-by: node-agent-suggester and the kubescape.io/suggest-request that produced them; only those ever get updated or withdrawn. Anything with a signature is yours. §7 and §8 do this review at mesh scale, blindly, so you can watch the agent alone.

7. Many to many

Six services, nine clients, two namespaces, headless Services in the mix: twenty-one edges nobody wrote down. Needs nothing from the steps above. The runner binds every running pod to a base with no network at all, so every flow in the mesh is unexpected while the agent already knows all of them from the shadows. It plays the reviewer: every half minute it asks every workload for a suggestion, and every proposal the agent answers with is signed as an admission fragment of its target bundle. Then it makes the binding durable, which rolls every workload out bound from its first packet, waits five minutes for silence, restarts everything again, and waits five more. The acceptance runs behind this lesson use ten.

OUT=~/mesh ~/gen-mesh.sh 6 9
OUT=~/mesh QUIET=300 SETTLE=60 ~/run-mesh.sh

Both keys come from ~/bob/example/redis/keys/: the vendor key signs the bases, the operator key signs the admission fragments, exactly as the trust policy of unit 3 demands. The report ends in ===== VERDICT PASS when every expected edge was accepted on the right port, nothing outside the topology was proposed except a workload talking to itself, and both windows were silent.

on chart duckling46 this one ends in VERDICT FAIL — read it as a bug report, §8 is the run that passes

All 21 edges were accepted, every target went quiet, and still five of the nine clients alerted through both windows:

edges accepted 21/21 (both=20 client=1 shadow=0), fragments signed 35
network alerts: window1=1376 window2=1681 (criterion: 0 and 0)
  mesh-a/cli-1-admit-delta: /:8004
  mesh-a/cli-6-admit-delta: /:53 /:8001 /:8003

cli-1 reaches svc-2, svc-4 and kube-dns; its shadow recorded svc-4 only, so its delta admits svc-4 only, and every later request answers delta-unchanged while R0011 keeps firing for kube-dns and svc-2. The same for cli-2 to cli-5: always kube-dns and the first target of the loop, never the others. cli-6 to cli-9 recorded everything and went silent. The five that fail all ran on the same node; the alerts prove the agent on that node saw the flows, the shadows prove it did not learn them. Learning and judging disagree on one node for the first connection after an idle sleep. That is the finding; this harness exists to produce it.

8. A real mesh — standalone

HTTPS in front, an API behind it, PostgreSQL, Redis, RabbitMQ over AMQP and its HTTP API, MariaDB, a psql cron, a redis-cli warmer, two exporters, Prometheus scraping four of them across three namespaces, every hop by DNS name. Seventeen edges, eight targets, nine clients, and four workloads that are both. Same runner, longer budget. This section needs nothing from units 1–2 or from §7: a freshly booted playground is enough, the keys and the scripts are installed with the lesson. Three commands, about half an hour:

OUT=~/mesh2 ~/gen-mesh2.sh
OUT=~/mesh2 QUIET=300 SETTLE=60 ~/run-mesh.sh
OUT=~/mesh2 ~/mesh-graph.sh

Solid arrow: the client declared it and the target saw it. Dashed: one witness only. Red: expected, not proposed. The picture is ~/mesh2/mesh2.png, drawn from ~/mesh2/matrix.tsv, which is what the agent proposed, not what the manifest says.

what one run of the acceptance harness looked like (chart duckling46, two-core playground, five-minute windows)

Bound 182 seconds after apply. Discovery over 254 seconds later with 26 fragments signed: the 17 edges, one self edge, and ten egress deltas. Seventeen of seventeen edges accepted, every one with both witnesses. Five minutes without a network alert, a rollout restart of all thirteen workloads, five more minutes without one. Twenty minutes in all. The agent ran the pipeline 212 times for the runner's requests; 200 of those answered "nothing new".

===== MESH ACCEPTANCE services=8 clients=9 edges=17
time: apply→bound 182s, bound→discovered 254s, total 1229s
edges accepted 17/17 (both=17 client=0 shadow=0), fragments signed 26
network alerts: window1=0 window2=0 (criterion: 0 and 0)
TARGET           CLIENT           PORT   EVIDENCE ACCEPTED-PORTS
data/mariadb     shop/worker      3306   both     3306
data/pg-exporter mon/prometheus   9187   both     9187
data/postgres    data/pg-exporter 5432   both     5432
data/postgres    shop/api         5432   both     5432
data/postgres    shop/cron        5432   both     5432
data/rabbit      mon/prometheus   15692  both     15692
data/rabbit      shop/api         15672  both     5672,15672
data/rabbit      shop/api         5672   both     5672,15672
data/rabbit      shop/worker      15672  both     5672,15672
data/rabbit      shop/worker      5672   both     5672,15672
data/redis       data/redis-exporter 6379   both     6379
data/redis       shop/api         6379   both     6379
data/redis       shop/cache-warmer 6379   both     6379
data/redis-exporter mon/prometheus   9121   both     9121
shop/api         mon/prometheus   8080   both     8080
shop/api         shop/web         8080   both     8080
shop/web         shop/loadgen     8443   both     8443

fragments not in the generated topology (self traffic is a workload reaching its own Service and is expected):
  data/rabbit-admit-data-rabbit (both, self traffic)

suggester load: runs by outcome since binding
    200 "outcome":"delta-empty"
     10 "outcome":"delta-written"
      2 "outcome":"delta-unchanged"

===== AUTO-HEAL COMPLETE: every edge accepted, no network alert in 300s before and after a full restart
===== VERDICT PASS

what this mesh taught the agent, in order

Suggestions are pulled, not pushed. The agent used to write proposals on a timer, with hold-downs and churn budgets to keep it from rewriting them. All of that is gone: one annotation on a live pod is a request, the agent answers it once, with one terminal Event naming the request id, and writes nothing nobody asked for. The runner above asks every round.
Suggestions are network only. The agent used to propose whole profiles. Everything it learned about files and processes is a portability problem of its own (temp tables, loader paths, credentials in argv) and is out of scope for now. It proposes peers and network deltas, nothing else.
Portable profiles speak in selectors. bobctl portable rewrites egress to a pod selector and a namespace selector and drops the Service reference. The rule engine matched a Service destination by address or by Service reference only, so every accepted profile alerted on all of its own egress. The engine now resolves a Service destination to the Service's selector. Without that, this harness could never go quiet.
A workload talking to itself is a real flow. RabbitMQ's epmd reaches its own pod on 4369. Skipping it as "unexpected" left the only alerts standing after the restart. It is proposed like every other peer and marked kubescape.io/peer-self.
The reviewer must read whole objects. The storage lists profiles as metadata only; a review that reads a spec from a list sees an empty one.
Scanning amplifies learning. The operator triggers a vulnerability relevancy scan on every profile change, and shadow profiles change every thirty seconds. On a two-core cluster that saturates the node. The installer of this lab turns vulnerability scanning off.
The TLS front was invisible on ingress, and it was the kernel path. Caddy opens its listeners as multipath TCP. From kernel 6.4 on, an MPTCP bind bypasses the hooks the socket enricher relies on, so the agent knew the packets arrived but could not say whose socket they were and discarded them. Egress needs no listener, so it stayed visible. The network gadget now reports such ingress and attributes it by network namespace.
A restart is the real test. Pod addresses get reused within seconds. A SYN for a Service whose old endpoint now belongs to another pod must not teach that pod an ingress it never served, and a pod that inherited an address must still resolve to itself. Both are handled; the second window above is the proof.

Note

Two keys, two roles: the vendor signed the redis base and is done. Everything added afterwards — the client SBOB, the ingress into redis, the id delta — is the operator's and carries operator.pem. The agent proposes; it holds no key and signs nothing. In whole-object mode any key verifies, so the roles here are convention; unit 3 makes them enforced.

Fragments — who may sign what

Units 1–2 signed whole objects: one profile, one signature, any key verifies. A bundle splits the profile into fragments, and a root-signed trust policy says per class who may sign and which spec paths the class may touch.

(a) fragments and their signers · (b) who may sign what · (c) the three checks · (d) composite or refusal

The chart ships a policy anchored to a published demo root; its class signers are the two demo keys in keys/. Keep the logs tab from unit 1 running.

the trust policy in force (what the agent mounted)

kubectl -n honey get cm node-agent-bundle-policy -o jsonpath='{.data.trust-policy\.json}' | python3 -c 'import sys,json; p=json.load(sys.stdin)["policy"]["classes"]; [print(c, "signers:", [s[:20]+"…" for s in v["signers"]], "paths:", v["allowedSpecPaths"]) for c,v in p.items()]'

base      signers: ['key:162a08d59135652ee0…']  paths: ['architectures', 'capabilities', 'execs', 'opens', 'syscalls', 'seccompProfile', 'endpoints', 'imageID', 'imageTag', 'rulePolicies', 'identifiedCallStacks', 'labelSelector']
overlay   signers: ['key:58fd31c363a8fac06d…']  paths: ['execs', 'opens']
admission signers: ['key:58fd31c363a8fac06d…']  paths: ['ingress', 'egress']

for k in vendor operator; do echo "$k key:$(openssl pkey -in keys/$k.pem -pubout -outform DER 2>/dev/null | sha256sum | cut -c1-64)"; done

vendor   key:162a08d59135652ee095ad2e0d927104c9e4eefe94f4ae844e7387ba1a7352d2
operator key:58fd31c363a8fac06d6eb596c2e8c63d5913b4dc4b971ee62579ec41fff78826

1. The vendor ships the base fragment

Same content as the unit-1 base, now labelled as a base fragment of bundle redis, signed with the vendor key:

export CP=containerprofiles.spdx.softwarecomposition.kubescape.io
cd ~/bob/example/redis && mkdir -p frags
kubectl create --dry-run=client -o json -f vendor/redis-base-signed.yaml | python3 -c '
import sys,json
d=json.load(sys.stdin)
d["metadata"]={"name":"redis-base","namespace":"redis","labels":{"signature.kubescape.io/bundle":"redis","signature.kubescape.io/fragment-class":"base"},"annotations":{"kubescape.io/managed-by":"User"}}
d["spec"]["ingress"]=None
json.dump(d,sys.stdout)' > frags/redis-base.json
~/sign-object sign --file frags/redis-base.json --key keys/vendor.pem --output frags/redis-base-signed.yaml
kubectl apply -f frags/redis-base-signed.yaml
kubectl -n redis delete $CP redis --ignore-not-found

The pod label kubescape.io/user-defined-profile: redis did not change — it now resolves to the bundle redis instead of an object named redis.

logs — one verified leaf, one root; and R0012 is back, because the base has no ingress

{"t":"…","level":"info","msg":"assembled signed bundle overlay"}
{"t":"…","rule":"R0012","alert":"Unexpected Ingress Network Traffic","ns":"redis","wl":"redis-master","msg":"Unexpected ingress network communication from: 10.42.0.49:6379 using TCP to: redis"}

kubectl -n honey logs ds/node-agent -c node-agent --all-pods=true --since=2m | grep -o '"msg":"assembled signed bundle overlay".*' | tail -1

"msg":"assembled signed bundle overlay","bundle":"redis","namespace":"redis","fragments":1,"root":"7c1e…"

2. The operator ships the admission fragment

The ingress both clients need — the entries the agent proposed in units 1 and 2 — as one admission fragment, signed with the operator key. If you skipped unit 2 there is no web-client; its entry then admits nothing and is harmless:

cat > frags/redis-client-ingress.json <<'EOF'
{"apiVersion":"spdx.softwarecomposition.kubescape.io/v1beta1","kind":"ContainerProfile",
 "metadata":{"name":"redis-client-ingress","namespace":"redis","labels":{"signature.kubescape.io/bundle":"redis","signature.kubescape.io/fragment-class":"admission"},"annotations":{"kubescape.io/managed-by":"User"}},
 "spec":{"ingress":[{"identifier":"redis-client","type":"internal",
   "namespaceSelector":{"matchLabels":{"kubernetes.io/metadata.name":"redis"}},
   "podSelector":{"matchLabels":{"app":"redis-client"}},
   "ports":[{"name":"TCP-6379","port":6379,"protocol":"TCP"}]},
   {"identifier":"web-client","type":"internal",
   "namespaceSelector":{"matchLabels":{"kubernetes.io/metadata.name":"client"}},
   "podSelector":{"matchLabels":{"app":"web-client"}},
   "ports":[{"name":"TCP-6379","port":6379,"protocol":"TCP"}]}]}}
EOF
~/sign-object sign --file frags/redis-client-ingress.json --key keys/operator.pem --output frags/redis-client-ingress-signed.yaml
kubectl apply -f frags/redis-client-ingress-signed.yaml

frags/redis-client-ingress-signed.yaml

kind: ContainerProfile
metadata:
  name: redis-client-ingress
  namespace: redis
  labels:
    signature.kubescape.io/bundle: redis
    signature.kubescape.io/fragment-class: admission
  annotations:
    kubescape.io/managed-by: User
    signature.kubescape.io/identity: local-key
    signature.kubescape.io/signature: MEQCIB…
    signature.kubescape.io/content: H4sIAAAA…
spec:
  ingress:
    - identifier: redis-client
      type: internal
      namespaceSelector:
        matchLabels: {kubernetes.io/metadata.name: redis}
      podSelector:
        matchLabels: {app: redis-client}
      ports: [{name: TCP-6379, port: 6379, protocol: TCP}]
    - identifier: web-client
      type: internal
      namespaceSelector:
        matchLabels: {kubernetes.io/metadata.name: client}
      podSelector:
        matchLabels: {app: web-client}
      ports: [{name: TCP-6379, port: 6379, protocol: TCP}]

logs — two leaves, a new root, R0012 stops for both clients

"msg":"assembled signed bundle overlay","bundle":"redis","namespace":"redis","fragments":2,"root":"a94f…"

3. The operator ships an overlay — and only execs/opens

cat > frags/redis-ops-overlay.json <<'EOF'
{"apiVersion":"spdx.softwarecomposition.kubescape.io/v1beta1","kind":"ContainerProfile",
 "metadata":{"name":"redis-ops-overlay","namespace":"redis","labels":{"signature.kubescape.io/bundle":"redis","signature.kubescape.io/fragment-class":"overlay"},"annotations":{"kubescape.io/managed-by":"User"}},
 "spec":{"execs":[{"path":"/usr/bin/df","args":["/usr/bin/df","-h"]}]}}
EOF
~/sign-object sign --file frags/redis-ops-overlay.json --key keys/operator.pem --output frags/redis-ops-overlay-signed.yaml
kubectl apply -f frags/redis-ops-overlay-signed.yaml
sleep 60
kubectl -n redis exec sts/redis-master -- df -h >/dev/null
kubectl -n redis exec sts/redis-master -- id >/dev/null

logs — three leaves; `df -h` is silent, `id` is not

"msg":"assembled signed bundle overlay","bundle":"redis","namespace":"redis","fragments":3,"root":"e21b…"

{"t":"…","rule":"R0001","alert":"Unexpected process launched","ns":"redis","wl":"redis-master","msg":"Unexpected process launched: id with PID …"}

4. Three ways to be refused

Wrong key for the class — the vendor signs an admission fragment:

~/sign-object sign --file frags/redis-client-ingress.json --key keys/vendor.pem --output frags/wrongkey.yaml
kubectl apply -f frags/wrongkey.yaml

Put the operator-signed admission fragment back before the next attempt, so each refusal is judged on its own and not in the shadow of the last one:

kubectl apply -f frags/redis-client-ingress-signed.yaml

Wrong paths for the class — an admission fragment smuggling an exec:

python3 -c '
import json
d=json.load(open("frags/redis-client-ingress.json")); d["metadata"]["name"]="redis-sneaky"
d["spec"]["execs"]=[{"path":"/bin/bash","args":["/bin/bash"]}]
json.dump(d,open("frags/sneaky.json","w"))'
~/sign-object sign --file frags/sneaky.json --key keys/operator.pem --output frags/sneaky-signed.yaml
kubectl apply -f frags/sneaky-signed.yaml

Tampered content — rewrite the signed bytes of the overlay:

kubectl -n redis get $CP redis-ops-overlay -o jsonpath='{.metadata.annotations.signature\.kubescape\.io/content}' \
  | python3 -c 'import sys,base64,gzip; d=gzip.decompress(base64.b64decode(sys.stdin.read())); print(base64.b64encode(gzip.compress(d.replace(b"/usr/bin/df", b"/bin/backdoor"))).decode())' \
  | xargs -I{} kubectl -n redis annotate $CP redis-ops-overlay --overwrite signature.kubescape.io/content={}

logs — each refusal names its reason; the workload keeps the last verified composite

signer not permitted for this fragment class: "redis-client-ingress"
fragment sets a spec path not permitted for its class: "redis-sneaky"
fragment signature does not verify (tampered): "redis-ops-overlay"

{"t":"…","level":"warn","msg":"signed bundle overlay refused: a verified member no longer verifies; keeping the last verified composite"}
{"t":"…","rule":"R1016","alert":"Signed profile tampered","ns":"redis","wl":"redis-master","msg":"…"}

Clean up the three and re-ship the good overlay:

kubectl -n redis delete $CP redis-sneaky
kubectl apply -f frags/redis-client-ingress-signed.yaml
kubectl apply -f frags/redis-ops-overlay-signed.yaml

5. Leaves are never rewritten

for f in redis-base redis-client-ingress redis-ops-overlay; do
  kubectl -n redis get $CP $f -o yaml > /tmp/leaf.yaml && ~/sign-object verify --file /tmp/leaf.yaml --strict=false >/dev/null && echo "leaf $f: OK"
done

leaf redis-base: OK
leaf redis-client-ingress: OK
leaf redis-ops-overlay: OK