In this chapter, we will be profiling applications using eBPF and you will see how this allows you to judge the quality and the risk that an installer or a piece of software brings to your system.

We will also compare different distributions (packagings or helm-charts) of the same functional software and how you can use profiling to inform your preferences, which distro to choose.

Last but not least, we will be looking at anomaly detection.

Kubescape Space PandaBear, according to legend, its name is Mo - all credit goes to Team Kubescape/Armosec

A. Clone the Repository & Install Prerequisites

cd ~
git clone https://github.com/k8sstormcenter/bobctl.git
cd bobctl
git checkout 62-retest-the-iximiuz-labs-after-folder-structure 
make storage kubescape-vendor

kubectl get pods -n honey -l app.kubernetes.io/instance=kubescape

laborant@dev-machine:~/honeycluster$ kubectl get pods -n honey -l app.kubernetes.io/instance=kubescape
NAME                        READY   STATUS    RESTARTS   AGE
kubescape-768648957-t52hf   1/1     Running   0          8m46s
kubevuln-77897b796c-vvs24   1/1     Running   0          8m46s
node-agent-zdxlh            1/1     Running   0          8m28s
operator-546d5845-p5hxd     1/1     Running   0          8m46s
storage-6cd74486bd-6tn6l    1/1     Running   0          8m46s

We now have kubescape nodeagent up and running, this is the eBPF component of kubescape (the project itself has many other components )

Step 2: Deploy the Application

Before using real applications, we start with deploying a well-known demo** called webapp that has:

• a) Desired functionality: it pings things.
• b) Undesired functionality: it is vulnerable to injection (runtime is compromised).
  • This is to mimic a CVE in your app.
• c) Tampering with the artefact: In module 2, we will additionally tamper with the artifact and make it create a backdoor (supply chain is compromised).
  • This is to mimic a SupplyChain corruption between vendor and you.

The following command installs the demo webapp:

cd ~/bobctl
make helm-install-no-bob

kubectl get pods -l app=webapp -o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}'

Step 3: Use the application during the learning phase

Let's trigger some of the Desired functionality: which in this case translates to a single ping

cd ~/bobctl
make fwd

After making sure the shell is not polluted with the fwd (you might have to hit enter)

curl localhost:8080/ping.php?ip=172.16.0.2

If that works, open a new tab new terminal and try some other IP adresses, if you so desire. Maybe note down the values you use, as you ll find them soon.

Step 4: Check that the learning is happening

kubectl logs -n honey -l app=node-agent -f -c node-agent

At the time of writing, we are anticipating the release of Kubescape 4.0, which will allow loading of externally recorded ApplicationProfiles. At KubeCon London 2025 a coffee between Constanze, David and Peter led to the initial idea of transferring fingerprints via OCI, Matthias quickly confirmed that the modifications to Kubescape would be doable, Vadim came up with the name "Bill of Behavior" shortly thereafter. And soon (time of writing is August 25), Pixie will be the first CNCF project with an SBoB. Other projects are planned/in the works. If you want one, please open an issue in the repo .

What is a (S)BoB?

A Software “Bill of Behavior” (BoB): a vendor-supplied profile detailing known benign runtime behaviors for software, designed to be distributed directly within OCI artifacts. Generated using eBPF, a BoB codifies expected syscalls, file access patterns, network communications, and capabilities. This empowers powerful, signature-less anomaly detection, allowing end-users to infer malicious activity or tampering in third-party software without the current burden of authoring and maintaining complex, custom security rules.

To solve the problem of distributing secure-by-default and up-to-date security rules for detection and defense.

Bill of Behavior: addresses the problem of distributing secure-by-default and up-to-date security rules for detection and defense- as it is supplied inside the application-packaging

There are two primary scenarios:

1. Runtime Anomalies

This scenario covers situations where a vulnerability (CVE or ZeroDay) is present in the app and it gets exploited.

2. Supply Chain Anomalies

This scenario covers threats originating from a compromised supply chain. For example:

• The artefact is not the one from the vendor (e.g. through typosquatting, registry poisoning, unauthorized access to artefactories )
• The vendor's supply chain got otherwise compromised (i.e. the artefact is signed by the vendor, but contains malicious pieces)

Thus the software, at runtime, will exhibit some form of behavior that was unintended by its creator. This could be a beacon, a backdoor, a cryptominer, a keylogger, a rootkit, altered images for defamation or any other form of nasty.

FAQ: Will this solve all of my security issues for now and ever more?

No, a BoB s main use is that an end-user inherits a behavior profile from those that understand the software-behavior: the people who created the software.This means, that an end users profiles are secure-by-default and can be kept up-to-date automatically.

There are still many types of attacks that hide within the benign behavior or can be possible for various reasons.

The Bill of Behavior Lifecycle

Diagram: Vendor Publication of a BoB

Benign Behavior

We assume that the full set of benign behaviour consists of the webapp performing a few pings internally to our cluster. Thus, we simply make the app execute a few such pings. This is not representative for all possible things that the webapp could do, but let's keep it simple for starters. In a production use-case, you d probably run various test-suites instead of a manual REST call.

Example to packaging the benign traffic tests into a software supply chain framework (such as Helm)

We envision for vendors to package verification tests into the software, ranging from simple unit-tests to more complex suites. For a sample webapp, we implemented a simple hook (which is essentially the same ping-test as above)

cd ~/bobctl
make helm-test

apiVersion: v1
kind: Pod
metadata:
  name: "{{ include "mywebapp.fullname" . }}-test-connection"
  labels:
    {{- include "mywebapp.labels" . | nindent 4 }}
    kubescape.io/ignore: "true"
  annotations:
    "helm.sh/hook": test
    "helm.sh/hook-delete-policy": hook-succeeded,hook-failed
spec:
  restartPolicy: Never
  containers:
    - name: curl
      image: curlimages/curl:8.7.1
      command:
        - /bin/sh
        - -c
        - |
          set -ex
          SERVICE="{{ include "mywebapp.fullname" . }}"
          NAMESPACE="{{ .Release.Namespace }}"
          PORT="{{ .Values.service.port }}"
          TARGET_IP="{{ .Values.bob.targetIp }}"
          URL="${SERVICE}.${NAMESPACE}.svc.cluster.local:${PORT}/ping.php?ip=${TARGET_IP}"
          RESPONSE=$(curl -s "$URL")
          echo "$RESPONSE"
          echo "$RESPONSE" | grep -q "Ping results for ${TARGET_IP}"
          echo "$RESPONSE" | grep -q "${TARGET_IP} ping statistics"

Deployment is ready. Running Helm tests...
helm test webapp -n webapp
NAME: webapp
LAST DEPLOYED: Thu Jul 17 11:35:26 2025
NAMESPACE: webapp
STATUS: deployed
REVISION: 1
TEST SUITE:     webapp-mywebapp-test-connection
Last Started:   Thu Jul 17 11:57:57 2025
Last Completed: Thu Jul 17 11:58:02 2025
Phase:          Succeeded
NOTES:
Your webapp-mywebapp application is now deployed.

Check again that the test was recorded

{"level":"info","ts":"2025-08-27T15:12:18Z","msg":"ContainerWatcher.containerCallback - processing container","containerID":"8a6005f0498cceb36b592d67aa3784e1128b61c5496cd9d1df4afe026a836565","namespace":"webapp","podName":"webapp-mywebapp-test-connection","callbackCount":10}

a little later, you will also see that the container has stopped processing as it has terminated

{"level":"info","ts":"2025-08-27T15:12:50Z","msg":"stopped recording container profile","reason":"container has terminated","containerID":"8a6005f0498cceb36b592d67aa3784e1128b61c5496cd9d1df4afe026a836565","containerName":"curl","podName":"webapp-mywebapp-test-connection","namespace":"webapp"}

One of the black-belt level disciplines in security is real-time anomaly detection. This is the story of how Kubescape makes it possible, obviously by using eBPF