Lesson  in  Software Bill of Behavior - vendor supplied runtime profile for tampering and anomaly detection

The agent suggests the client's SBOB — you sign it

A vendor-signed redis, an unprotected client, and node-agent proposing the client's portable profile on its own. Prove it equals bobctl, sign it, bind it, watch the alerts stop.

Suggest, prove, sign, bind

0. One terminal for logs

Open a second tab logs, run this once and leave it:

kubectl -n honey logs ds/node-agent -c node-agent --all-pods=true -f | sed -u 's/^\[[^]]*\] //' | jq -Rc --unbuffered 'fromjson? | if .RuleID then {t:.time, rule:.RuleID, alert:.BaseRuntimeMetadata.alertName, ns:.RuntimeK8sDetails.namespace, wl:.RuntimeK8sDetails.workloadName, msg:.message} else {t:.ts, level, msg} end'
ready when you see this — the suggester line and every tracer on every node
{"t":"…","level":"info","msg":"suggester enabled"}
{"t":"…","level":"info","msg":"Started tracer"}
{"t":"…","level":"info","msg":"ContainerWatcher started successfully"}
{"t":".","level":"info","msg":"RogueArtifact writer wired: discoverable signoff state enabled"}
{"t":".","level":"info","msg":"queue processor started"}
{"t":".","level":"info","msg":"Successfully verified object signature"}
{"t":".","level":"info","msg":"signed bundle overlays enabled in alert mode"}
{"t":".","level":"warn","msg":"signed bundle overlays anchored to the PUBLISHED DEMO root key: this authenticates nothing, mount a real root before relying on signatures"}
{"t":".","level":"warn","msg":"rule signing DISABLED: the trust policy in force carries no ruleClasses; ANY Rules object in ANY namespace will load without a signature check"}
{"t":".","level":"info","msg":"trust policy in force"}

1. The vendor ships redis with a signed SBOB — no client is known yet

_redis with SBOB signed by vendor _

cd ~/bob/example/redis
export CP=containerprofiles.spdx.softwarecomposition.kubescape.io
kubectl create ns redis
kubectl apply -f vendor/redis-base-signed.yaml
helm install redis oci://registry-1.docker.io/bitnamicharts/redis --version 27.0.18 -n redis --set architecture=standalone --set auth.enabled=false --set image.repository=bitnami/redis --set image.digest=sha256:08863c2c3f4e051fb6139b38fa223e9c13be5033326a59bead182860d899bf98 \
  --set 'master.podLabels.kubescape\.io/user-defined-profile=redis'
logs
{"t":".","level":"warn","msg":"signed fragment stored spec diverges from the signed content: the stored spec is display-only and is NOT enforced; enforcement uses the signed content"}
{"t":".","level":"info","msg":"adopted user-authored ContainerProfile as authoritative base"}
{"t":"…","level":"info","msg":"container has a user defined profile"}
{"t":"…","level":"info","msg":"Successfully verified object signature"}

The reason for the tamper alert is that during the first tick, the operator hasnt reconciled

the vendor's SBOB, abridged — 15 execs · 79 opens · 8 capabilities · no ingress
kind: ContainerProfile
metadata:
  name: redis
  namespace: redis
  annotations:
    kubescape.io/managed-by: User
    signature.kubescape.io/identity: local-key
    signature.kubescape.io/issuer: local
    signature.kubescape.io/timestamp: "1790366987"
    signature.kubescape.io/signature: MEUCIQDd5Ba9Hg2iSh2OAO8w…
    signature.kubescape.io/certificate: LS0tLS1CRUdJTiBDRVJUSUZJ…
    signature.kubescape.io/content: H4sIAAAAAAAA/7xZ73LiOBJ/…
spec:
  matchLabels: {app.kubernetes.io/component: master, app.kubernetes.io/instance: redis, app.kubernetes.io/name: redis}
  capabilities: [CAP_CHOWN, CAP_DAC_OVERRIDE, CAP_NET_ADMIN, CAP_SETGID, CAP_SETUID, CAP_SYS_ADMIN, …]
  execs:
    - {args: [/opt/bitnami/redis/bin/redis-cli, -h, localhost, -p, "6379", ping], path: /opt/bitnami/redis/bin/redis-cli}
    - {args: [/opt/bitnami/redis/bin/redis-server, ⋯⋯], path: /opt/bitnami/redis/bin/redis-server}
    - {args: [/health/ping_liveness_local.sh, ⋯⋯], path: /usr/bin/bash}
    - …
  opens:
    - {flags: [O_RDONLY], path: /data}
    - {flags: [O_APPEND, O_CREAT, O_WRONLY], path: /data/appendonlydir/appendonly.aof.1.incr.aof}
    - …
  ingress: null

2. A client appears — unprotected

no profile, no trust: the client is a rogue artefact

sed '/kubescape.io\/user-defined-profile/d' client.yaml | kubectl apply -f -
kubectl -n redis rollout status deploy/redis-client --timeout=120s
Alerts R1017 Rogue artefact, R0012 Unexpected Ingress Network Traffic on redis for every connection, R0040 Unexpected process arguments
{"t":"…","rule":"R1017","alert":"Rogue artefact","ns":"redis","wl":"redis-client","msg":"Container 'client' in namespace 'redis' has no bound profile — an unmanaged/unknown container is running in the cluster"}
{"t":"…","rule":"R0012","alert":"Unexpected Ingress Network Traffic","ns":"redis","wl":"redis-master","msg":"Unexpected ingress network communication from: 10.42.0.49:6379 using TCP to: redis"}
{"t":"…","rule":"R0040","alert":"Unexpected process arguments","ns":"redis","wl":"redis-client","msg":"Unexpected process arguments: redis-cli with PID 76287"}

3. The agent suggests a profile (~2 min)

watch -n5 "kubectl -n redis get $CP -l 'kubescape.io/profile-role=suggestion,!kubescape.io/delta-of'"
kubectl -n redis get events --field-selector reason=SBoBSuggested -o custom-columns=POD:.involvedObject.name,MSG:.message
discovered vs. portable
shadow — what was learned
suggestion — what is proposed

125 execs · 13 opens · 63 syscalls · 2 egress

spec:
  execs:
    - {path: /bin/busybox, args: [/bin/sh, -c, "i=0\nwhile true; do\n  redis-cli -h redis-master -p 6379 SET \"k:$i\" \"v:$i\" >/dev/null\n  i=$((i+1)); sleep 2\ndone\n"]}
    - {path: /bin/busybox, args: [sleep, "2"]}
    - {path: /pause, args: [/pause]}
    - {path: /usr/local/bin/redis-cli, args: [/usr/local/bin/redis-cli, -h, redis-master, -p, "6379", SET, k:32, v:32]}
    - {path: /usr/local/bin/redis-cli, args: [/usr/local/bin/redis-cli, -h, redis-master, -p, "6379", SET, k:33, v:33]}
    - {path: /usr/local/bin/redis-cli, args: [/usr/local/bin/redis-cli, -h, redis-master, -p, "6379", SET, k:34, v:34]}
    - … 119 more redis-cli lines, one per SET
  opens:
    - {path: /etc/resolv.conf, flags: [O_CLOEXEC, O_LARGEFILE, O_RDONLY]}
    - {path: /etc/hosts, flags: [O_CLOEXEC, O_LARGEFILE, O_RDONLY]}
    - {path: /lib/libssl.so.3, flags: [O_CLOEXEC, O_LARGEFILE, O_RDONLY]}
    - {path: /proc/sys/vm/overcommit_memory, flags: [O_CLOEXEC, O_RDONLY]}
    - … 9 more
  syscalls: [accept4, bind, brk, clone, close, connect, dup2, execve, exit_group, fcntl, getpid, ioctl, … 51 more]
  egress:
    - identifier: f8ed3623…
      type: internal
      serviceRefName: redis-master
      serviceRefNamespace: redis
      podSelector:
        matchLabels: {app.kubernetes.io/component: master, app.kubernetes.io/instance: redis, app.kubernetes.io/name: redis}
      ports: [{name: TCP-6379, port: 6379, protocol: TCP}]
    - identifier: f14f2fcf…
      type: internal
      serviceRefName: kube-dns
      serviceRefNamespace: kube-system
      namespaceSelector:
        matchLabels: {kubernetes.io/metadata.name: kube-system}
      podSelector:
        matchLabels: {k8s-app: kube-dns}
      ports: [{name: UDP-53, port: 53, protocol: UDP}]

5 execs (was 125) · 13 opens (was 13) · 0 syscalls (63 removed) · 2 egress (was 2, now by identity)

spec:
  architectures: [amd64]
  imageID: docker.io/library/redis@sha256:858f009f…
  matchLabels: {app: redis-client}
  execs:
    - {path: /bin/busybox, args: [/bin/sh, ⋯⋯]}
    - {path: /bin/busybox, args: [/bin/sleep, ⋯⋯]}
    - {path: /bin/busybox, args: [sleep, ⋯⋯]}
    - {path: /pause, args: [/pause]}
    - {path: /usr/local/bin/redis-cli, args: [/usr/local/bin/redis-cli, -h, redis-master, -p, "6379", SET, ⋯, ⋯]}
  opens:
    - {path: /etc/resolv.conf, flags: [O_CLOEXEC, O_LARGEFILE, O_RDONLY]}
    - {path: /etc/hosts, flags: [O_CLOEXEC, O_LARGEFILE, O_RDONLY]}
    - {path: /lib/libssl.so.3, flags: [O_CLOEXEC, O_LARGEFILE, O_RDONLY]}
    - {path: /proc/sys/vm/overcommit_memory, flags: [O_CLOEXEC, O_RDONLY]}
    - … 9 more
  egress:
    - identifier: f8ed3623…
      type: internal
      serviceRefName: redis-master
      serviceRefNamespace: redis
      podSelector:
        matchLabels: {app.kubernetes.io/component: master, app.kubernetes.io/instance: redis, app.kubernetes.io/name: redis}
      ports: [{name: TCP-6379, port: 6379, protocol: TCP}]
    - identifier: f14f2fcf…
      type: internal
      serviceRefName: kube-dns
      serviceRefNamespace: kube-system
      namespaceSelector:
        matchLabels: {kubernetes.io/metadata.name: kube-system}
      podSelector:
        matchLabels: {k8s-app: kube-dns}
      ports: [{name: UDP-53, port: 53, protocol: UDP}]

4. You sign it with your key and bind it

no patience? the suggested profile, as a file

yolo/client-sbob.json is exactly what the agent suggests for this client. Use it instead of the first three lines below:

cp yolo/client-sbob.json client-sbob.json
~/sign-object generate-keypair --output operator.pem
N=$(kubectl -n redis get $CP -l 'kubescape.io/profile-role=suggestion,!kubescape.io/delta-of' -o jsonpath='{.items[0].metadata.name}')
kubectl -n redis get $CP "$N" -o json | python3 -c '
import sys,json
d=json.load(sys.stdin)
d["metadata"]={"name":"redis-client","namespace":"redis","annotations":{"kubescape.io/managed-by":"User"}}
json.dump(d,sys.stdout)' > client-sbob.json
~/bobctl validate --file client-sbob.json
~/sign-object sign --file client-sbob.json --key operator.pem --output client-sbob-signed.yaml
kubectl apply -f client-sbob-signed.yaml
kubectl -n redis patch deploy redis-client -p '{"spec":{"template":{"metadata":{"labels":{"kubescape.io/user-defined-profile":"redis-client"}}}}}'
kubectl -n redis rollout status deploy/redis-client --timeout=120s
logs
{"t":"…","level":"info","msg":"Successfully verified object signature"}
{"t":"…","level":"info","msg":"adopted user-authored ContainerProfile as authoritative base"}
client-sbob-signed.yaml, abridged
kind: ContainerProfile
metadata:
  name: redis-client
  namespace: redis
  annotations:
    kubescape.io/managed-by: User
    signature.kubescape.io/identity: local-key
    signature.kubescape.io/issuer: local
    signature.kubescape.io/signature: MEUCIQClx5bZ…
    signature.kubescape.io/content: H4sIAAAAAAAA/7RW…
spec:
  architectures: [amd64]
  matchLabels: {app: redis-client}
  execs:
    - {args: [/bin/sh, ⋯⋯], path: /bin/busybox}
    - {args: [/usr/local/bin/redis-cli, -h, redis-master, -p, "6379", SET, ⋯, ⋯], path: /usr/local/bin/redis-cli}
    - …
  egress:
    - {serviceRefName: redis-master, serviceRefNamespace: redis, ports: [{name: TCP-6379, port: 6379, protocol: TCP}], …}
    - {serviceRefName: kube-dns, serviceRefNamespace: kube-system, ports: [{name: UDP-53, port: 53, protocol: UDP}], …}

5. The agent already proposed the ingress — the operator signs it into redis

The client is bound, redis is bound. The agent now looks at both: where the client talks to, and what redis actually received. One proposal, two witnesses:

kubectl -n redis get $CP -l kubescape.io/delta-of=redis -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.annotations.kubescape\.io/evidence}{"\n"}{end}'
redis-delta-overlay
redis-peer-redis-redis-client   both

The first line is redis's own drift (one open the vendor did not record). The second is the client.

the proposal — the agent's ingress entry for redis, not yet signed
kind: ContainerProfile
metadata:
  name: redis-peer-redis-redis-client
  namespace: redis
  labels: {kubescape.io/profile-role: suggestion, kubescape.io/delta-of: redis, kubescape.io/peer-workload: redis-client}
  annotations: {kubescape.io/evidence: both, kubescape.io/created-by: node-agent-suggester}
spec:
  matchLabels: {app.kubernetes.io/component: master, app.kubernetes.io/instance: redis, app.kubernetes.io/name: redis}
  ingress:
    - identifier: redis-redis-client
      type: internal
      namespaceSelector:
        matchLabels: {kubernetes.io/metadata.name: redis}
      podSelector:
        matchLabels: {app: redis-client}
      ports: [{name: TCP-6379, port: 6379, protocol: TCP}]

client = the client's suggestion says it sends there. shadow = redis saw it arrive. both = the two agree.

The vendor shipped redis without knowing this client; admitting it is the operator's decision. Merge the proposal into the vendor SBOB and re-sign with the operator key:

kubectl -n redis get $CP redis -o json | jq --argjson add "$(kubectl -n redis get $CP redis-peer-redis-redis-client -o json | jq '.spec.ingress')" '.spec.ingress = ((.spec.ingress // []) + $add) | {apiVersion, kind, metadata: {name: .metadata.name, namespace: .metadata.namespace, annotations: {"kubescape.io/managed-by": "User"}}, spec}' > redis-healed.json
~/sign-object sign --file redis-healed.json --key operator.pem --output redis-healed-signed.yaml && kubectl apply -f redis-healed-signed.yaml
no proposal yet, or no patience? two shortcuts

yolo/redis-healed.json is the redis SBOB with the client ingress already merged (it also admits unit 2's web-client):

~/sign-object sign --file yolo/redis-healed.json --key operator.pem --output redis-healed-signed.yaml && kubectl apply -f redis-healed-signed.yaml

Or derive the entry yourself from the client's suggestion, the way it was done before the agent proposed it:

~/overlay.sh operator.pem redis "$(kubectl -n redis get $CP -l 'kubescape.io/profile-role=suggestion,!kubescape.io/delta-of' -o jsonpath='{.items[0].metadata.name}')"
logs — redis re-verified, R0012 stops within a minute
{"t":"…","level":"info","msg":"Successfully verified object signature"}

6. Detection, then the correction

The client keeps writing SET k:N. The profile learned two minutes of it as literal arguments, so every new key is an R0040 now:

logs — judged against your profile, not the shadow
{"t":"…","rule":"R0040","alert":"Unexpected process arguments","ns":"redis","wl":"redis-client","msg":"Unexpected process arguments: redis-cli with PID 84895 argv=/usr/local/bin/redis-cli -h redis-master -p 6379 SET k:133 v:133"}

The agent has already computed what ends it, one entry per binary:

kubectl -n redis get $CP -l kubescape.io/delta-of=redis-client -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.annotations.kubescape\.io/suggestion-version}{"\n"}{end}'
redis-client-delta-overlay   1
the delta — what the signed profile does not grant
spec:
  matchLabels: {app: redis-client}
  execs:
    - {path: /usr/local/bin/redis-cli, args: [/usr/local/bin/redis-cli, ⋯⋯]}

⋯⋯ is any arguments. The literal entries stay; this one entry makes them irrelevant.

Merge it into the client profile and re-sign, same move as in step 5:

kubectl -n redis get $CP redis-client -o json | jq --argjson add "$(kubectl -n redis get $CP redis-client-delta-overlay -o json | jq '{execs: (.spec.execs // []), opens: (.spec.opens // [])}')" '.spec.execs = ((.spec.execs // []) + $add.execs) | .spec.opens = ((.spec.opens // []) + $add.opens) | {apiVersion, kind, metadata: {name: .metadata.name, namespace: .metadata.namespace, annotations: {"kubescape.io/managed-by": "User"}}, spec}' > client-sbob.json
~/sign-object sign --file client-sbob.json --key operator.pem --output client-sbob-signed.yaml && kubectl apply -f client-sbob-signed.yaml

Quiet. The proposal disappears on its own: the profile admits it now. Do something the profile never saw:

kubectl -n redis exec deploy/redis-client -- id
logs — and a minute later, a new proposal
{"t":"…","rule":"R0001","alert":"Unexpected process launched","ns":"redis","wl":"redis-client","msg":"Unexpected process launched: id with PID 83923"}
{"t":"…","rule":"R0004","alert":"Linux Capabilities Anomalies in container","ns":"redis","wl":"redis-client","msg":"Unexpected capability used: CAP_SYS_ADMIN in syscall read with PID 82970"}
kubectl -n redis get $CP redis-client-delta-overlay -o jsonpath='{.spec.execs}{"\n"}'

7. Tamper

sed 's#signature.kubescape.io/signature: ME#signature.kubescape.io/signature: MF#' client-sbob-signed.yaml | kubectl apply -f -
logs (within a minute)
{"t":"…","level":"warn","msg":"Object signature verification failed"}
{"t":"…","level":"warn","msg":"user-defined ContainerProfile signature mismatch (tamper detected)"}
kubectl apply -f client-sbob-signed.yaml
Important

kubectl get $CP lists come back with every spec empty — storage serves metadata only for lists. Take the name from the list, then kubectl get $CP <name> for the object. Signing a list item signs an empty profile: the agent enforces nothing and everything alerts. bobctl validate catches it before you sign.

Autodiscovered — the agent proposes, the operator signs

1. A different client, in its own namespace

It fetches a web page over TLS and writes a timestamp into redis across namespaces:

cd ~/bob/example/redis
export CP=containerprofiles.spdx.softwarecomposition.kubescape.io
kubectl create ns client
kubectl apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-client
  namespace: client
  labels: {app: web-client}
spec:
  replicas: 1
  selector:
    matchLabels: {app: web-client}
  template:
    metadata:
      labels: {app: web-client}
    spec:
      containers:
        - name: client
          image: redis:7-alpine
          command: ["/bin/sh", "-c"]
          args:
            - |
              i=0
              while true; do
                wget -qO /tmp/quote.txt https://labs.iximiuz.com/ || true
                redis-cli -h redis-master.redis.svc.cluster.local -p 6379 SET "quote:$i" "$(date -u +%s)" >/dev/null
                i=$((i+1)); sleep 3
              done
EOF
kubectl -n client rollout status deploy/web-client --timeout=120s
logs — R1017 on the client; R0012 on redis, because nobody allowed this client
{"t":"…","rule":"R1017","alert":"Rogue artefact","ns":"client","wl":"web-client","msg":"Container 'client' in namespace 'client' has no bound profile — an unmanaged/unknown container is running in the cluster"}
{"t":"…","rule":"R0012","alert":"Unexpected Ingress Network Traffic","ns":"redis","wl":"redis-master","msg":"Unexpected ingress network communication from: 10.42.1.23:6379 using TCP to: redis"}

2. Two proposals appear (~2–3 min)

kubectl get $CP -A -l kubescape.io/profile-role=suggestion -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{"\t"}{.metadata.labels.kubescape\.io/delta-of}{"\t"}{.metadata.annotations.kubescape\.io/evidence}{"\n"}{end}'
client/replicaset-web-client-767858fd8c-client-4ae4-9b79-suggested
redis/redis-delta-overlay              redis
redis/redis-peer-redis-redis-client    redis   both
redis/redis-peer-client-web-client     redis   both

The first is the client's own profile. The last is new: nobody wrote it, nobody ran a script. The agent saw the client's egress name redis, saw redis's own shadow receive it, and proposed the admission entry on redis, across namespaces.

kubectl -n client get events --field-selector reason=SBoBPeerProposed -o jsonpath='{.items[0].note}{"\n"}'
redis/redis-master receives traffic from client/web-client that its profile redis does not admit. Proposal redis-peer-client-web-client (evidence: client). Review, sign and apply it as an admission fragment of redis, or merge it into redis.
discovered vs. portable — the client's suggestion
shadow — what was learned
suggestion — what gets signed

96 execs · 25 opens · 75 syscalls · 6 egress. Addresses, one per resolution; commands, one per invocation:

spec:
  execs:
    - {path: /bin/busybox, args: [wget, -qO, /tmp/quote.txt, https://labs.iximiuz.com/]}
    - {path: /usr/bin/ssl_client, args: [ssl_client, -s, "3", -n, labs.iximiuz.com]}
    - {path: /bin/busybox, args: [sleep, "3"]}
    - {path: /usr/local/bin/redis-cli, args: [/usr/local/bin/redis-cli, -h, redis-master.redis.svc.cluster.local, -p, "6379", SET, quote:17, "1790417063"]}
    - {path: /usr/local/bin/redis-cli, args: [/usr/local/bin/redis-cli, -h, redis-master.redis.svc.cluster.local, -p, "6379", SET, quote:18, "1790417066"]}
    - … 90 more
  opens:
    - {path: /etc/resolv.conf, flags: [O_CLOEXEC, O_LARGEFILE, O_RDONLY]}
    - {path: /etc/ssl/certs/ca-certificates.crt, flags: [O_RDONLY]}
    - {path: /tmp/quote.txt, flags: [O_CREAT, O_LARGEFILE, O_TRUNC, O_WRONLY]}
    - … 22 more
  syscalls: [accept4, bind, brk, clone, close, connect, getsockopt, ioctl, poll, … 66 more]
  egress:
    - identifier: 8e352065…
      type: external
      dnsNames: [labs.iximiuz.com.]
      ipAddress: 104.26.15.92
      ports: [{name: TCP-443, port: 443, protocol: TCP}]
    - identifier: 6eccf30d…
      type: external
      dnsNames: [labs.iximiuz.com.]
      ipAddress: 104.26.14.92
      ports: [{name: TCP-443, port: 443, protocol: TCP}]
    - … redis-master, kube-dns

11 execs · 25 opens · 0 syscalls · 4 egress · 1 endpoint · 4 capabilities:

spec:
  architectures: [amd64]
  imageID: docker.io/library/redis@sha256:858f009f…
  matchLabels: {app: web-client}
  capabilities: [CAP_SETGID, CAP_SETPCAP, CAP_SETUID, CAP_SYS_ADMIN]
  execs:
    - {path: /bin/busybox, args: [/bin/sh, ⋯⋯]}
    - {path: /bin/busybox, args: [wget, ⋯⋯]}
    - {path: /bin/busybox, args: [sleep, ⋯⋯]}
    - {path: /usr/bin/ssl_client, args: [ssl_client, ⋯⋯]}
    - {path: /usr/local/bin/redis-cli, args: [/usr/local/bin/redis-cli, -h, redis-master.redis.svc.cluster.local, -p, "6379", SET, ⋯, ⋯]}
    - … 7 more
  opens:
    - {path: /etc/resolv.conf, flags: [O_CLOEXEC, O_LARGEFILE, O_RDONLY]}
    - {path: /etc/ssl/certs/ca-certificates.crt, flags: [O_RDONLY]}
    - {path: /tmp/quote.txt, flags: [O_CREAT, O_LARGEFILE, O_TRUNC, O_WRONLY]}
    - … 22 more
  egress:
    - identifier: tuned-labs.iximiuz.com
      type: external
      dnsNames: [labs.iximiuz.com]
      ipAddresses: [104.26.14.92]
      ports: [{name: TCP-443, port: 443, protocol: TCP}]
    - identifier: 0b4b36aa…
      type: internal
      serviceRefName: redis-master
      serviceRefNamespace: redis
      podSelector:
        matchLabels: {app.kubernetes.io/component: master, app.kubernetes.io/instance: redis, app.kubernetes.io/name: redis}
      ports: [{name: TCP-6379, port: 6379, protocol: TCP}]
    - identifier: f14f2fcf…
      type: internal
      serviceRefName: kube-dns
      serviceRefNamespace: kube-system
      ports: [{name: UDP-53, port: 53, protocol: UDP}]

3. The operator signs the client and binds it

N=$(kubectl -n client get $CP -l 'kubescape.io/profile-role=suggestion,!kubescape.io/delta-of' -o jsonpath='{.items[0].metadata.name}')
kubectl -n client get $CP "$N" -o json | jq '{apiVersion, kind, metadata: {name: "web-client", namespace: .metadata.namespace, annotations: {"kubescape.io/managed-by": "User"}}, spec}' > web-client-sbob.json
~/bobctl validate --file web-client-sbob.json
~/sign-object sign --file web-client-sbob.json --key operator.pem --output web-client-signed.yaml && kubectl apply -f web-client-signed.yaml
kubectl -n client patch deploy web-client -p '{"spec":{"template":{"metadata":{"labels":{"kubescape.io/user-defined-profile":"web-client"}}}}}'
kubectl -n client rollout status deploy/web-client --timeout=120s
no patience? the suggested profile, as a file
~/sign-object sign --file yolo/web-client-sbob.json --key operator.pem --output web-client-signed.yaml && kubectl apply -f web-client-signed.yaml
kubectl -n client patch deploy web-client -p '{"spec":{"template":{"metadata":{"labels":{"kubescape.io/user-defined-profile":"web-client"}}}}}'

4. The operator admits the client into redis

Same move as unit 1, with the entry the agent proposed:

kubectl -n redis get $CP redis -o json | jq --argjson add "$(kubectl -n redis get $CP redis-peer-client-web-client -o json | jq '.spec.ingress')" '.spec.ingress = ((.spec.ingress // []) + $add) | {apiVersion, kind, metadata: {name: .metadata.name, namespace: .metadata.namespace, annotations: {"kubescape.io/managed-by": "User"}}, spec}' > redis-healed.json
~/sign-object sign --file redis-healed.json --key operator.pem --output redis-healed-signed.yaml && kubectl apply -f redis-healed-signed.yaml
what redis admits now — two clients, two namespaces
spec:
  ingress:
    - identifier: redis-redis-client
      type: internal
      namespaceSelector:
        matchLabels: {kubernetes.io/metadata.name: redis}
      podSelector:
        matchLabels: {app: redis-client}
      ports: [{name: TCP-6379, port: 6379, protocol: TCP}]
    - identifier: client-web-client
      type: internal
      namespaceSelector:
        matchLabels: {kubernetes.io/metadata.name: client}
      podSelector:
        matchLabels: {app: web-client}
      ports: [{name: TCP-6379, port: 6379, protocol: TCP}]
logs — verified, adopted, R0012 stops
{"t":"…","level":"info","msg":"Successfully verified object signature"}
{"t":"…","level":"info","msg":"adopted user-authored ContainerProfile as authoritative base"}

5. Quiet — until something new happens, and then a proposal

kubectl -n client exec deploy/web-client -- id
logs
{"t":"…","rule":"R0001","alert":"Unexpected process launched","ns":"client","wl":"web-client","msg":"Unexpected process launched: id with PID 83923"}
{"t":"…","rule":"R0002","alert":"Files Access Anomalies in container","ns":"client","wl":"web-client","msg":"Unexpected file access detected: id with PID 83923 to /etc/group"}

The alert is the detection. About a minute later the agent adds the correction, as a proposal you can sign or ignore:

kubectl -n client get $CP -l kubescape.io/delta-of=web-client -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.annotations.kubescape\.io/suggestion-version}{"\n"}{end}'
web-client-delta-overlay    1
the delta — only what the signed profile does not grant
spec:
  matchLabels: {app: web-client}
  execs:
    - {path: /bin/busybox, args: [/usr/bin/id, ⋯⋯]}
  opens:
    - {path: /etc/group, flags: [O_CLOEXEC, O_LARGEFILE, O_RDONLY]}
    - {path: /etc/passwd, flags: [O_CLOEXEC, O_LARGEFILE, O_RDONLY]}

Merge and re-sign as in step 4 if id belongs in the profile. Leave it and it expires after a day. Run id fifty more times and the proposal does not change: it holds the first version until the behaviour is stable, and never more than a few replacements an hour.

6. The agent never overwrites what you signed

sleep 120
kubectl get $CP -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{"\t"}{.metadata.annotations.signature\.kubescape\.io/identity}{"\n"}{end}' | grep -v $'\t$'
client/web-client     local-key
redis/redis           local-key
redis/redis-client    local-key

Proposals carry kubescape.io/created-by: node-agent-suggester and only those ever get updated or withdrawn. Anything with a signature is yours.

hands-free? the acceptor signs every suggestion it sees
pkill -f "[a]cceptor.sh"; NS_SKIP="honey kube-system redis" nohup ~/acceptor.sh operator.pem > acceptor.log 2>&1 &
tail -f acceptor.log

It signs, applies and binds each unsigned suggestion with operator.pem, and merges the ingress into redis itself. A bare loop, no judgement: the proposals above are what it should be reading instead.

Note

Two keys, two roles: the vendor signed the redis base and is done. Everything added afterwards — the client SBOB, the ingress into redis, the id delta — is the operator's and carries operator.pem. The agent proposes; it holds no key and signs nothing. In whole-object mode any key verifies, so the roles here are convention; unit 3 makes them enforced.

Fragments — who may sign what

Units 1–2 signed whole objects: one profile, one signature, any key verifies. A bundle splits the profile into fragments, and a root-signed trust policy says per class who may sign and which spec paths the class may touch.

(a) fragments and their signers · (b) who may sign what · (c) the three checks · (d) composite or refusal

The chart ships a policy anchored to a published demo root; its class signers are the two demo keys in keys/. Keep the logs tab from unit 1 running.

the trust policy in force (what the agent mounted)
kubectl -n honey get cm node-agent-bundle-policy -o jsonpath='{.data.trust-policy\.json}' | python3 -c 'import sys,json; p=json.load(sys.stdin)["policy"]["classes"]; [print(c, "signers:", [s[:20]+"…" for s in v["signers"]], "paths:", v["allowedSpecPaths"]) for c,v in p.items()]'
base      signers: ['key:162a08d59135652ee0…']  paths: ['architectures', 'capabilities', 'execs', 'opens', 'syscalls', 'seccompProfile', 'endpoints', 'imageID', 'imageTag', 'rulePolicies', 'identifiedCallStacks', 'labelSelector']
overlay   signers: ['key:58fd31c363a8fac06d…']  paths: ['execs', 'opens']
admission signers: ['key:58fd31c363a8fac06d…']  paths: ['ingress', 'egress']
for k in vendor operator; do echo "$k key:$(openssl pkey -in keys/$k.pem -pubout -outform DER 2>/dev/null | sha256sum | cut -c1-64)"; done
vendor   key:162a08d59135652ee095ad2e0d927104c9e4eefe94f4ae844e7387ba1a7352d2
operator key:58fd31c363a8fac06d6eb596c2e8c63d5913b4dc4b971ee62579ec41fff78826

1. The vendor ships the base fragment

Same content as the unit-1 base, now labelled as a base fragment of bundle redis, signed with the vendor key:

export CP=containerprofiles.spdx.softwarecomposition.kubescape.io
cd ~/bob/example/redis && mkdir -p frags
kubectl create --dry-run=client -o json -f vendor/redis-base-signed.yaml | python3 -c '
import sys,json
d=json.load(sys.stdin)
d["metadata"]={"name":"redis-base","namespace":"redis","labels":{"signature.kubescape.io/bundle":"redis","signature.kubescape.io/fragment-class":"base"},"annotations":{"kubescape.io/managed-by":"User"}}
d["spec"]["ingress"]=None
json.dump(d,sys.stdout)' > frags/redis-base.json
~/sign-object sign --file frags/redis-base.json --key keys/vendor.pem --output frags/redis-base-signed.yaml
kubectl apply -f frags/redis-base-signed.yaml
kubectl -n redis delete $CP redis --ignore-not-found

The pod label kubescape.io/user-defined-profile: redis did not change — it now resolves to the bundle redis instead of an object named redis.

logs — one verified leaf, one root; and R0012 is back, because the base has no ingress
{"t":"…","level":"info","msg":"assembled signed bundle overlay"}
{"t":"…","rule":"R0012","alert":"Unexpected Ingress Network Traffic","ns":"redis","wl":"redis-master","msg":"Unexpected ingress network communication from: 10.42.0.49:6379 using TCP to: redis"}
kubectl -n honey logs ds/node-agent -c node-agent --all-pods=true --since=2m | grep -o '"msg":"assembled signed bundle overlay".*' | tail -1
"msg":"assembled signed bundle overlay","bundle":"redis","namespace":"redis","fragments":1,"root":"7c1e…"

2. The operator ships the admission fragment

The ingress the client needs — the same content the agent derived in unit 1 — as an admission fragment, signed with the operator key:

cat > frags/redis-client-ingress.json <<'EOF'
{"apiVersion":"spdx.softwarecomposition.kubescape.io/v1beta1","kind":"ContainerProfile",
 "metadata":{"name":"redis-client-ingress","namespace":"redis","labels":{"signature.kubescape.io/bundle":"redis","signature.kubescape.io/fragment-class":"admission"},"annotations":{"kubescape.io/managed-by":"User"}},
 "spec":{"ingress":[{"identifier":"redis-client","type":"internal",
   "namespaceSelector":{"matchLabels":{"kubernetes.io/metadata.name":"redis"}},
   "podSelector":{"matchLabels":{"app":"redis-client"}},
   "ports":[{"name":"TCP-6379","port":6379,"protocol":"TCP"}]}]}}
EOF
~/sign-object sign --file frags/redis-client-ingress.json --key keys/operator.pem --output frags/redis-client-ingress-signed.yaml
kubectl apply -f frags/redis-client-ingress-signed.yaml
frags/redis-client-ingress-signed.yaml
kind: ContainerProfile
metadata:
  name: redis-client-ingress
  namespace: redis
  labels:
    signature.kubescape.io/bundle: redis
    signature.kubescape.io/fragment-class: admission
  annotations:
    kubescape.io/managed-by: User
    signature.kubescape.io/identity: local-key
    signature.kubescape.io/signature: MEQCIB…
    signature.kubescape.io/content: H4sIAAAA…
spec:
  ingress:
    - identifier: redis-client
      type: internal
      namespaceSelector:
        matchLabels: {kubernetes.io/metadata.name: redis}
      podSelector:
        matchLabels: {app: redis-client}
      ports: [{name: TCP-6379, port: 6379, protocol: TCP}]
logs — two leaves, a new root, R0012 stops
"msg":"assembled signed bundle overlay","bundle":"redis","namespace":"redis","fragments":2,"root":"a94f…"

3. The operator ships an overlay — and only execs/opens

cat > frags/redis-ops-overlay.json <<'EOF'
{"apiVersion":"spdx.softwarecomposition.kubescape.io/v1beta1","kind":"ContainerProfile",
 "metadata":{"name":"redis-ops-overlay","namespace":"redis","labels":{"signature.kubescape.io/bundle":"redis","signature.kubescape.io/fragment-class":"overlay"},"annotations":{"kubescape.io/managed-by":"User"}},
 "spec":{"execs":[{"path":"/usr/bin/df","args":["/usr/bin/df","-h"]}]}}
EOF
~/sign-object sign --file frags/redis-ops-overlay.json --key keys/operator.pem --output frags/redis-ops-overlay-signed.yaml
kubectl apply -f frags/redis-ops-overlay-signed.yaml
sleep 60
kubectl -n redis exec sts/redis-master -- df -h >/dev/null
kubectl -n redis exec sts/redis-master -- id >/dev/null
logs — three leaves; `df -h` is silent, `id` is not
"msg":"assembled signed bundle overlay","bundle":"redis","namespace":"redis","fragments":3,"root":"e21b…"
{"t":"…","rule":"R0001","alert":"Unexpected process launched","ns":"redis","wl":"redis-master","msg":"Unexpected process launched: id with PID …"}

4. Three ways to be refused

Wrong key for the class — the vendor signs an admission fragment:

~/sign-object sign --file frags/redis-client-ingress.json --key keys/vendor.pem --output frags/wrongkey.yaml
kubectl apply -f frags/wrongkey.yaml

Wrong paths for the class — an admission fragment smuggling an exec:

python3 -c '
import json
d=json.load(open("frags/redis-client-ingress.json")); d["metadata"]["name"]="redis-sneaky"
d["spec"]["execs"]=[{"path":"/bin/bash","args":["/bin/bash"]}]
json.dump(d,open("frags/sneaky.json","w"))'
~/sign-object sign --file frags/sneaky.json --key keys/operator.pem --output frags/sneaky-signed.yaml
kubectl apply -f frags/sneaky-signed.yaml

Tampered content — rewrite the signed bytes of the overlay:

kubectl -n redis get $CP redis-ops-overlay -o jsonpath='{.metadata.annotations.signature\.kubescape\.io/content}' \
  | python3 -c 'import sys,base64,gzip; d=gzip.decompress(base64.b64decode(sys.stdin.read())); print(base64.b64encode(gzip.compress(d.replace(b"/usr/bin/df", b"/bin/backdoor"))).decode())' \
  | xargs -I{} kubectl -n redis annotate $CP redis-ops-overlay --overwrite signature.kubescape.io/content={}
logs — each refusal names its reason; the workload keeps the last verified composite
signer not permitted for this fragment class: "redis-client-ingress"
fragment sets a spec path not permitted for its class: "redis-sneaky"
fragment signature does not verify (tampered): "redis-ops-overlay"
{"t":"…","level":"warn","msg":"signed bundle overlay refused: a verified member no longer verifies; keeping the last verified composite"}
{"t":"…","rule":"R1016","alert":"Signed profile tampered","ns":"redis","wl":"redis-master","msg":"…"}

Clean up the three and re-ship the good overlay:

kubectl -n redis delete $CP redis-sneaky
kubectl apply -f frags/redis-client-ingress-signed.yaml
kubectl apply -f frags/redis-ops-overlay-signed.yaml

5. Leaves are never rewritten

for f in redis-base redis-client-ingress redis-ops-overlay; do
  kubectl -n redis get $CP $f -o yaml > /tmp/leaf.yaml && ~/sign-object verify --file /tmp/leaf.yaml --strict=false >/dev/null && echo "leaf $f: OK"
done
leaf redis-base: OK
leaf redis-client-ingress: OK
leaf redis-ops-overlay: OK
Important

The keys in keys/ and the root the chart trusts are published demo material — the agent says so at boot (anchored to the PUBLISHED DEMO root key: this authenticates nothing). A real deployment mounts its own root public key, ships its own policy, and keeps every private key off the cluster.