Adaptive Kubernetes Sovereign Security Operations Center
Multimessenger Security: An adaptive kubernetes souvereign SOC composed from Musketeers CNCF eBPF Projects
The linux kernel through eBPF offers to unify the disparate fields security and observability through shared data structures. We show how a K8s Security Operations Center, organically composed of established eBPF projects (CNCF Kubescape, Pixie and Tetragon) can see signals that the individuals cannot.
We explain how we achieve both a comprehensive baseline and use independent signals to dial up/down coverage as suspicious indicators surface. The mutual independence of signals from across processes, file system, and network activity achieves a high signal-to-noise, enabling manageable data volumes and facilitating selective forensic storage.
An adaptive Kubernetes Sovereign Security Operations Center based on eBPF
Acknowledgement: This work is funded in part by Netidee grant 7918/2025
Lets deploy our composite eBPF SOC - uncompromised
1) the newest release of Kubescape
We first install the nodeagent component of CNCF Kubescape, with a specific config that defineds Kubescape's learning period and some exclusion rules.
cd /home/laborant
git clone https://github.com/k8sstormcenter/bob.git
cd bob
make kubescape
Install Kubescape Nodeagent
2) a custom version of Pixie
As the second component, we install a fork of CNCF Pixie and the Pixie Cli px. This fork has the special feature, that it can read and write from/to an external database.
cd /home/laborant
git clone https://github.com/k8sstormcenter/honeycluster.git
cd honeycluster
Install (a fork of) Pixie incl its SBoB
The below commands install a normal pixie and connect it to the UI, then glob-deploy the fork and the BoBs.
cd /home/laborant/honeycluster
./honeystack/pixie/pixie_vizier/sbob/bobapply.sh
export PX_CLOUD_ADDR=getcosmic.ai
px auth login
px deploy
./honeystack/pixie/pixie_vizier/sbob/bobapply.sh
kubectl apply -f honeystack/pixie/pixie_vizier/entlein_pixieNov3.yaml
./honeystack/pixie/pixie_vizier/sbob/bobapply.sh
3) a columnar database (Clickhouse) and log forwarding (Vector)
We now deploy the hyperDX chart for Clickhouse, this chart has lots of components which we dont use, but for the KubeConNA SecurityCon Talk,
it is a perfect showcase.
cd ~/honeycluster
make clickhouse vector
Install Clickhouse DB and enable log forwarding
Lets adaptively react
Now, our detection stack is all setup. Kubescape is evaluation the differences between an SBoB and the runtime behavior.
Now we infect our stack
We pretend our database stack has been supply chain compromised
cd /home/laborant/honeycluster
make update-clickhouse
so this "update" actually pulls in our rootkit
We mock compromise clickhouse app server
Story 1: Kubescape detects an anomaly
Extract the metadata from the "possibly infected" process
We look at all current anomalies (whatever diffs from the SBoB)
px run -f /home/laborant/honeycluster/analysis/px_clickhouse/kubescape/observe.pxl
laborant@dev-machine:honeycluster$ px run -f /home/laborant/honeycluster/analysis/px_clickhouse/kubescape/observe.pxl
Pixie CLI
*******************************
* ENV VARS
* PX_CLOUD_ADDR=getcosmic.ai
*******************************
Table ID: kubescape_pods
MESSAGE PODNAME NAMESPACE TIMESTAMP
Unexpected system call: chmod hyperdx-hdx-oss-v2-otel-collector-64c698f998-87qf6 click 1.761668491518777e+18
Unexpected system call: mkdir hyperdx-hdx-oss-v2-otel-collector-64c698f998-87qf6 click 1.7616684918184387e+18
Unexpected system call: select hyperdx-hdx-oss-v2-mongodb-56dc967bcf-8kskr click 1.7616685275085857e+18
Unexpected io_uring operation detected: Epoll control (opcode=29) flags=0x0 in next-server (v. hyperdx-hdx-oss-v2-app-6bc79cdd9b-rqfwq click 1.7616685188072218e+18
Unexpected io_uring operation detected: Epoll control (opcode=29) flags=0x0 in next-server (v. hyperdx-hdx-oss-v2-app-6bc79cdd9b-rqfwq click 1.7616685188072428e+18
Unexpected io_uring operation detected: Epoll control (opcode=29) flags=0x0 in next-server (v. hyperdx-hdx-oss-v2-app-6bc79cdd9b-rqfwq click 1.7616685188071826e+18
Unexpected system call: io_uring_register hyperdx-hdx-oss-v2-app-client-775864d696-fhb4m click 1.7616690892466696e+18
Now we focus on the io_uring_register syscall , to understand if it s a legit software update or not.
2025-10-19 16:13:3 {"alertName":"Unexpected system call","arguments":{"syscall":"io_uring_register"},"infectedPID":61910,"md5Hash":"8cb5d303c3f5643c6fdaa3c450b6c449","profileMetadata":{"error":{},"failOnProfile":true,"type":0},"severity":1,"sha1Hash":"a6a18998930a8aff2bd0f8e99fcf14d0d7bfd372","size":"4.6 MB","timestamp":"2025-10-19T16:13:30.389130871Z","trace":{},"uniqueID":"edb92c8b30983d8ee0343a8d5a1da64d"} empty R0003 {"clusterName":"bobexample","containerID":"b83d9b8a6798acad4571cb35b0c3413a93447875d5be328de9f76bd28a182a40","containerName":"client","hostNetwork":false,"namespace":"cure","podName":"curing-client-7c6d4bbfb8-fssx5","podNamespace":"cure","workloadKind":"Deployment","workloadName":"curing-client","workloadNamespace":"cure"} {"containerID":"b83d9b8a6798acad4571cb35b0c3413a93447875d5be328de9f76bd28a182a40","processTree":{"cmdline":"./client","comm":"client","path":"/app/client","pcomm":"containerd-shim","pid":61910,"ppid":61713,"startTime":"0001-01-01T00:00:00Z"}} {"k8s":{"containerName":"client","namespace":"cure","node":"node-01","owner":{},"podLabels":{"app":"curing-client","kubescape.io/user-defined-profile":"curing-client","pod-template-hash":"7c6d4bbfb8"},"podName":"curing-client-7c6d4bbfb8-fssx5"},"runtime":{"containerId":"b83d9b8a6798acad4571cb35b0c3413a93447875d5be328de9f76bd28a182a40","runtimeName":"containerd"},"timestamp":1760890410389130871,"type":"normal"} error Unexpected system call: io_uring_register Unexpected system call 2025-10-19T16:13:30Z
Lets pull up the full SBoB and compare,
Before the "update" our clickhouse app-server was a npm-application which required the two following syscalls at runtime:
#SBoB for clickhouse app
apiVersion: spdx.softwarecomposition.kubescape.io/v1beta1
kind: ApplicationProfile
metadata:
name: app
namespace: click
...
processAllowed:
- next-server
- node⋯
- node
- npm exec concur-⋯
- npm-⋯
- npm
...
syscalls:
- io_uring_enter
- io_uring_setup
If you re doing this at home, feel free to check the whole SBoB:
cat /home/laborant/honeycluster/honeystack/clickhouse/clickhouseapp.yaml
Now, we export the relevant filters from the anomaly.
export PODNAME=$(px run -f /home/laborant/honeycluster/analysis/px_clickhouse/kubescape/display_kubescape.pxl | grep register| grep '"podName"' | awk -F'"podName":"' '{print $2}' | awk -F'"' '{print $1}' | head -n1)
export namespace=$(px run -f /home/laborant/honeycluster/analysis/px_clickhouse/kubescape/display_kubescape.pxl |grep register| grep '"namespace":"' | awk -F'"namespace":"' '{print $2}' | awk -F'"' '{print $1}' | head -n1)
export COMM=$(px run -f /home/laborant/honeycluster/analysis/px_clickhouse/kubescape/display_kubescape.pxl |grep register| grep '"namespace":"' | awk -F'"comm":"' '{print $2}' | awk -F'"' '{print $1}' | head -n1)
echo "There is a suspicious pod : $PODNAME in namespace: $namespace running comm: $COMM"
Pixie CLI
*******************************
* ENV VARS
* PX_CLOUD_ADDR=getcosmic.ai
*******************************
There is a suspicious pod : hyperdx-hdx-oss-v2-app-client-775864d696-fhb4m in namespace: click running comm: client .
Story 2: Pixie starts writing to (Forensic)DB
Pixie can now write to Clickhouse adaptively
All data in pixie that mentions the pod hyperdx-hdx-oss-v2-app-client-775864d696-fhb4m in namespace click involved in the io_uring_register syscall gets written to DB.
Story 3: Pixie is asked for the last 10 min of network traffic from the pod
We can detect it in dns
There is some network traffic: it looks very "spikey"
First, we query the historical records in the official pixie tables. Filtered on the metadata from the above anomaly
cat <<EOF > pod_tcp_conns.pxl
import px
def tcp_conns():
df = px.DataFrame(table='conn_stats')
df.namespace = df.ctx['namespace']
df.pod = df.ctx['pod']
df = df[px.contains(df.pod, "${PODNAME}")]
df = df[not df.remote_addr == '127.0.0.1']
return df
px.display(tcp_conns())
EOF
px run -f pod_tcp_conns.pxl
Now, we're seeing both DNS and TCP traffic, but the TCP is not recognized as a known protocol (Protocol = 0 , means unknown). That is unusual
laborant@dev-machine:honeycluster$ px run -f pod_tcp_conns.pxl
Pixie CLI
Table ID: output
TIME REMOTE ADDR REMOTE PORT TRACE ROLE ADDR FAMILY PROTOCOL SSL CONN OPEN CONN CLOSE CONN ACTIVE BYTES SENT BYTES RECV NAMESPACE POD
2025-10-28 16:42:12 10.43.124.187 8888 1 1 0 false 198 198 0 181 KiB 16.8 KiB click click/hyperdx-hdx-oss-v2-app-client-775864d696-fhb4m
2025-10-28 16:42:12 10.43.0.10 53 1 1 6 false 1582 1582 0 134 KiB 272 KiB click click/hyperdx-hdx-oss-v2-app-client-775864d696-fhb4m
2025-10-28 16:42:22 10.43.124.187 8888 1 1 0 false 201 201 0 184 KiB 17.1 KiB click click/hyperdx-hdx-oss-v2-app-client-775864d696-fhb4m
2025-10-28 16:42:22 10.43.0.10 53 1 1 6 false 1606 1606 0 136 KiB 276 KiB click click/hyperdx-hdx-oss-v2-app-client-775864d696-fhb4m
2025-10-28 16:42:32 10.43.124.187 8888 1 1 0 false 204 204 0 186 KiB 17.3 KiB click click/hyperdx-hdx-oss-v2-app-client-775864d696-fhb4m
2025-10-28 16:42:32 10.43.0.10 53 1 1 6 false 1629 1629 0 138 KiB 280 KiB click click/hyperdx-hdx-oss-v2-app-client-775864d696-fhb4m
2025-10-28 16:42:42 10.43.124.187 8888 1 1 0 false 207 207 0 189 KiB 17.6 KiB click click/hyperdx-hdx-oss-v2-app-client-775864d696-fhb4m
2025-10-28 16:42:42 10.43.0.10 53 1 1 6 false 1653 1653 0 140 KiB 284 KiB click click/hyperdx-hdx-oss-v2-app-client-775864d696-fhb4m
Network -> where is it talking to?
cat <<EOF > network.pxl
import px
def network():
df = px.DataFrame('conn_stats')
# Filter on namespace.
df = df[df.ctx['namespace'] == '${namespace}']
# Filter for client side requests.
df = df[df.trace_role == 1]
df.pod = df.ctx['pod']
# Filter out any non k8s sources.
df = df[df.pod != '']
# Find the time window
time_window = df.agg(
time_min=('time_', px.min),
time_max=('time_', px.max),
)
time_window.time_delta = px.DurationNanos(time_window.time_max - time_window.time_min)
time_window = time_window.drop(['time_min', 'time_max'])
df = df.groupby(['pod', 'upid', 'remote_addr']).agg(
bytes_sent_min=('bytes_sent', px.min),
bytes_sent_max=('bytes_sent', px.max),
bytes_recv_min=('bytes_recv', px.min),
bytes_recv_max=('bytes_recv', px.max),
)
df.bytes_sent = df.bytes_sent_max - df.bytes_sent_min
df.bytes_recv = df.bytes_recv_max - df.bytes_recv_min
df.bytes_total = df.bytes_sent + df.bytes_recv
df = df.drop(['bytes_sent_max', 'bytes_sent_min', 'bytes_recv_max', 'bytes_recv_min'])
df.from_entity = df.pod
localhost_ip_regexp = r'127\.0\.0\.[0-9]+'
df.is_remote_addr_localhost = px.regex_match(localhost_ip_regexp, df.remote_addr)
df.to_entity = px.select(df.is_remote_addr_localhost,
df.pod,
px.nslookup(df.remote_addr))
# Filter out entities as specified by the user.
df = df[px.contains(df.from_entity, "${PODNAME}")]
# Since there may be multiple processes per pod,
# perform an additional aggregation to consolidate those into one entry.
df = df.groupby(['from_entity', 'to_entity']).agg(
bytes_sent=('bytes_sent', px.sum),
bytes_recv=('bytes_recv', px.sum),
bytes_total=('bytes_total', px.sum),
)
# Add time_delta to every row. Use a join to do this.
# Future syntax will support: df.time_delta = time_window.at[0, 'time_delta']
df.join_key = 1
time_window.join_key = 1
df = df.merge(time_window, how='inner', left_on='join_key', right_on='join_key')
df = df.drop(['join_key_x', 'join_key_y'])
# Compute as rates.
df.bytes_sent = df.bytes_sent / df.time_delta
df.bytes_recv = df.bytes_recv / df.time_delta
df.bytes_total = df.bytes_total / df.time_delta
df = df.drop(['time_delta'])
return df
px.display(network())
EOF
px run -f network.pxl
px run -f network.pxl
Pixie CLI
*******************************
* ENV VARS
* PX_CLOUD_ADDR=getcosmic.ai
*******************************
Table ID: output
FROM ENTITY TO ENTITY BYTES SENT BYTES RECV BYTES TOTAL
click/hyperdx-hdx-oss-v2-app-client-fhb4m kube-dns.kube-system.svc.cluster.local 122 B/sec 248 B/sec 371 B/sec
click/hyperdx-hdx-oss-v2-app-client-fhb4m curing-server-service.cure.svc.cluster.local 165 B/sec 15.0 B/sec 180 B/sec
Stacktrace
We zoom in to the compromised pod in the compromised namespace
Can see curing in :
** stack_trace_beta in the end shows us what is happening -> we can download the github packages by amit -> and any decent analyst at this point has found the attacker (or rather the software they smuggled in)
cat <<EOF > stacktrace.pxl
import px
def stacktraces():
df = px.DataFrame(table='stack_traces.beta')
df.namespace = df.ctx['namespace']
df.pod = df.ctx['pod']
df.container = df.ctx['container']
df.cmdline = df.ctx['cmdline']
df.node = px.Node(px._exec_hostname())
df = df[px.contains(df.pod, '${PODNAME}')]
df = df[df.namespace == '${namespace}']
return df
px.display(stacktraces())
EOF
px run -f stacktrace.pxl
Here now we have the full stacktrace of the suspicous process:
Stacktrace
TIME STACK TRACE ID STACK TRACE COUNT NAMESPACE POD CONTAINER CMDLINE NODE
2025-10-22 15:58:21.425844934 +0000 UTC 6777 runtime.goexit.abi0;main.main.gowrap2;github.com/amitschendel/curing/pkg/client.(*CommandPuller).Run;github.com/amitschendel/curing/pkg/client.(*CommandPuller).connectReadAndProcess;github.com/amitschendel/curing/pkg/client.(*CommandPuller).connect;net.DialTimeout;net.(*Dialer).DialContext;net.(*sysDialer).dialParallel;net.(*sysDialer).dialSerial;net.(*sysDialer).dialSingle;net.(*sysDialer).dialTCP;net.(*sysDialer).doDialTCPProto;net.internetSocket;net.socket;net.(*netFD).dial;net.(*netFD).connect;syscall.Connect;syscall.connect;syscall.Syscall;internal/runtime/syscall.Syscall6;[k] entry_SYSCALL_64_after_hwframe;[k] kretprobe_trampoline;[k] kretprobe_trampoline;[k] trampoline_handler;[k] __kretprobe_trampoline_handler;[k] _raw_spin_unlock_irqrestore 1 click click/curing-client-7c6d4bbfb8-rbkgv client ./client node-01
2025-10-22 15:59:51.590700167 +0000 UTC 7215 runtime.goexit.abi0;net.(*Resolver).goLookupIPCNAMEOrder.func3.gowrap1;net.(*Resolver).goLookupIPCNAMEOrder.func3.1;net.(*Resolver).tryOneName;net.(*Resolver).exchange;net.dnsPacketRoundTrip;net.(*conn).Write;net.(*netFD).Write;internal/poll.(*FD).Write;syscall.write;syscall.Syscall;internal/runtime/syscall.Syscall6;[k] entry_SYSCALL_64_after_hwframe;[k] kretprobe_trampoline;[k] __x64_sys_write;[k] ksys_write;[k] vfs_write;[k] new_sync_write;[k] sock_write_iter;[k] __sock_sendmsg;[k] inet_sendmsg;[k] udp_sendmsg;[k] udp_send_skb;[k] ip_send_skb;[k] ip_output;[k] ip_finish_output;[k] __ip_finish_output;[k] ip_finish_output2;[k] neigh_resolve_output;[k] dev_queue_xmit;[k] __dev_queue_xmit;[k] dev_hard_start_xmit;[k] xmit_one.constprop.0;[k] dev_queue_xmit_nit;[k] packet_rcv;[k] bpf_prog_332f10ee454943d0_ig_trace_dns;[k] bpf_skb_event_output;[k] bpf_event_output;[k] perf_event_output;[k] perf_output_end;[k] irq_work_queue;[k] __irq_work_queue_local;[k] arch_irq_work_raise;[k] native_apic_msr_write 1 click click/curing-client-7c6d4bbfb8-rbkgv client ./client node-01
2025-10-22 16:01:21.71209691 +0000 UTC 7585 runtime.goexit.abi0;net.(*Resolver).goLookupIPCNAMEOrder.func3.gowrap1;net.(*Resolver).goLookupIPCNAMEOrder.func3.1;net.(*Resolver).tryOneName;net.(*Resolver).exchange;net.(*Resolver).dial;net.(*Dialer).DialContext;net.(*sysDialer).dialParallel;net.(*sysDialer).dialSerial;net.(*sysDialer).dialSingle;net.(*sysDialer).dialUDP;net.internetSocket;net.socket;net.(*netFD).dial;net.(*netFD).connect;syscall.Connect;syscall.connect;syscall.Syscall;internal/runtime/syscall.Syscall6;[k] entry_SYSCALL_64_after_hwframe;[k] kretprobe_trampoline;[k] kretprobe_trampoline;[k] trampoline_handler;[k] __kretprobe_trampoline_handler;[k] _raw_spin_unlock_irqrestore 1 click click/curing-client-7c6d4bbfb8-rbkgv client ./client node-01
2025-10-22 16:01:21.71209691 +0000 UTC 7612 runtime.goexit.abi0;net.(*Resolver).goLookupIPCNAMEOrder.func3.gowrap1;net.(*Resolver).goLookupIPCNAMEOrder.func3.1;runtime.chansend1;runtime.chansend;runtime.send;runtime.systemstack.abi0;runtime.send.goready.func1;runtime.ready;runtime.runqput 1 click click/curing-client-7c6d4bbfb8-rbkgv client ./client node-01
2025-10-22 16:02:51.793386797 +0000 UTC 7904 ;[k] ret_from_fork;[k] io_wqe_worker 1 click click/curing-client-7c6d4bbfb8-rbkgv client ./client node-01
2025-10-22 16:02:51.793386797 +0000 UTC 7906 runtime.goexit.abi0;net.(*Resolver).goLookupIPCNAMEOrder.func3.gowrap1;net.(*Resolver).goLookupIPCNAMEOrder.func3.1;net.(*Resolver).tryOneName;net.(*Resolver).exchange;net.dnsPacketRoundTrip;runtime.makeslice;runtime.mallocgc;runtime.memclrNoHeapPointers 1 click click/curing-client-7c6d4bbfb8-rbkgv client ./client node-01
2025-10-22 16:02:51.793386797 +0000 UTC 7950 runtime.goexit.abi0;net.(*Resolver).goLookupIPCNAMEOrder.func3.gowrap1;net.(*Resolver).goLookupIPCNAMEOrder.func3.1;net.(*Resolver).tryOneName;net.(*Resolver).exchange;net.dnsPacketRoundTrip;net.(*conn).Read;net.(*netFD).Read;internal/poll.(*FD).Read;syscall.read;syscall.Syscall;internal/runtime/syscall.Syscall6;[k] entry_SYSCALL_64_after_hwframe;[k] kretprobe_trampoline;[k] kretprobe_trampoline;[k] trampoline_handler;[k] __kretprobe_trampoline_handler;[k] _raw_spin_unlock_irqrestore 1 click click/curing-client-7c6d4bbfb8-rbkgv client ./client node-01
2025-10-22 16:04:21.864827833 +0000 UTC 8255 runtime.goexit.abi0;net.(*Resolver).goLookupIPCNAMEOrder.func3.gowrap1;net.(*Resolver).goLookupIPCNAMEOrder.func3.1;net.(*Resolver).tryOneName;net.(*Resolver).exchange;net.dnsPacketRoundTrip;net.(*conn).Write;net.(*netFD).Write;internal/poll.(*FD).Write;syscall.write;syscall.Syscall;internal/runtime/syscall.Syscall6;[k] entry_SYSCALL_64_after_hwframe;[k] kretprobe_trampoline;[k] __x64_sys_write;[k] ksys_write;[k] vfs_write;[k] new_sync_write;[k] sock_write_iter;[k] __sock_sendmsg;[k] inet_sendmsg;[k] udp_sendmsg;[k] udp_send_skb;[k] ip_send_skb;[k] ip_output;[k] ip_finish_output;[k] __ip_finish_output;[k] ip_finish_output2;[k] __local_bh_enable_ip;[k] do_softirq;[k] do_softirq_own_stack;[k] asm_call_sysvec_on_stack;[k] __softirqentry_text_start;[k] net_rx_action;[k] process_backlog;[k] __netif_receive_skb;[k] __netif_receive_skb_one_core;[k] __netif_receive_skb_core.constprop.0;[k] br_handle_frame;[k] br_nf_pre_routing;[k] br_nf_pre_routing_finish;[k] br_nf_hook_thresh;[k] br_handle_frame_finish;[k] br_pass_frame_up;[k] netif_receive_skb;[k] __netif_receive_skb;[k] __netif_receive_skb_one_core;[k] ip_rcv;[k] nf_hook_slow;[k] ip_sabotage_in;[k] ip_rcv_finish;[k] ip_forward;[k] ip_forward_finish;[k] ip_output;[k] ip_finish_output;[k] __ip_finish_output;[k] ip_finish_output2;[k] neigh_hh_output;[k] dev_queue_xmit;[k] __dev_queue_xmit;[k] dev_hard_start_xmit;[k] xmit_one.constprop.0;[k] vxlan_xmit;[k] vxlan_xmit_one;[k] vxlan_get_route 1 click click/curing-client-7c6d4bbfb8-rbkgv client ./client node-01
2025-10-22 16:07:22.072140627 +0000 UTC 00000005-0000-7c8c-0000-0000000367e6 8864 0x000000c0002c8060;runtime.send;runtime.systemstack.abi0;runtime.send.goready.func1;runtime.ready;runtime.wakep;runtime.startm;runtime.notewakeup;runtime.mstart.abi0;runtime.mstart0;runtime.mstart1;runtime.sysmon;runtime.nanotime1.abi0;[k] asm_sysvec_apic_timer_interrupt;[k] sysvec_apic_timer_interrupt;[k] irq_exit_rcu;[k] do_softirq_own_stack;[k] asm_call_sysvec_on_stack;[k] __softirqentry_text_start;[k] rcu_core_si;[k] rcu_core;[k] note_gp_changes;[k] _raw_spin_unlock_irqrestore 1 click click/curing-client-7c6d4bbfb8-rbkgv client ./client node-01
2025-10-22 16:10:22.30384519 +0000 UTC 9729 runtime.goexit.abi0;net.(*Resolver).goLookupIPCNAMEOrder.func3.gowrap1;net.(*Resolver).goLookupIPCNAMEOrder.func3.1;net.(*Resolver).tryOneName;net.(*Resolver).exchange;net.dnsPacketRoundTrip;net.(*conn).Read;net.(*netFD).Read;internal/poll.(*FD).Read;syscall.read;syscall.Syscall;internal/runtime/syscall.Syscall6;[k] entry_SYSCALL_64_after_hwframe;[k] kretprobe_trampoline;[k] __x64_sys_read;[k] ksys_read;[k] vfs_read;[k] new_sync_read;[k] sock_read_iter;[k] sock_recvmsg;[k] inet_recvmsg;[k] udp_recvmsg;[k] __skb_recv_udp;[k] __skb_try_recv_from_queue 1 click click/curing-client-7c6d4bbfb8-rbkgv client ./client node-01
2025-10-22 16:10:22.30384519 +0000 UTC 9756 runtime.goexit.abi0;net.(*Resolver).goLookupIPCNAMEOrder.func3.gowrap1;net.(*Resolver).goLookupIPCNAMEOrder.func3.1;net.(*Resolver).tryOneName;net.(*Resolver).exchange;net.(*Resolver).dial;net.(*Dialer).DialContext;net.(*sysDialer).dialParallel;net.(*sysDialer).dialSerial;net.(*sysDialer).dialSingle;net.(*sysDialer).dialUDP;net.internetSocket;net.socket;net.sysSocket;syscall.Socket;syscall.socket;syscall.RawSyscall;internal/runtime/syscall.Syscall6;[k] entry_SYSCALL_64_after_hwframe;[k] do_syscall_64;[k] __x64_sys_socket;[k] __sys_socket;[k] kretprobe_trampoline;[k] sock_alloc;[k] new_inode_pseudo;[k] alloc_inode;[k] inode_init_always 1 click click/curing-client-7c6d4bbfb8-rbkgv client ./client node-01
2025-10-22 16:10:22.30384519 +0000 UTC 9760 runtime.goexit.abi0;net.(*Resolver).goLookupIPCNAMEOrder.func3.gowrap1;net.(*Resolver).goLookupIPCNAMEOrder.func3.1;net.(*Resolver).tryOneName;net.(*Resolver).exchange;net.(*Resolver).dial;net.(*Dialer).DialContext;net.(*sysDialer).dialParallel;net.(*sysDialer).dialSerial;net.(*sysDialer).dialSingle;net.(*sysDialer).dialUDP;net.internetSocket;net.socket;net.(*netFD).dial;net.(*netFD).connect;syscall.Connect;syscall.connect;syscall.Syscall;internal/runtime/syscall.Syscall6;[k] entry_SYSCALL_64_after_hwframe;[k] kretprobe_trampoline;[k] kretprobe_trampoline;[k] trampoline_handler;[k] __kretprobe_trampoline_handler;[k] _raw_spin_unlock_irqrestore 1 click click/curing-client-7c6d4bbfb8-rbkgv client ./client node-01
2025-10-22 16:11:52.406689359 +0000 UTC 10036 runtime.goexit.abi0;net.(*Resolver).goLookupIPCNAMEOrder.func3.gowrap1;net.(*Resolver).goLookupIPCNAMEOrder.func3.1;net.(*Resolver).tryOneName;net.(*Resolver).exchange;net.dnsPacketRoundTrip;net.(*conn).Read;net.(*netFD).Read;internal/poll.(*FD).Read;syscall.read;syscall.Syscall;internal/runtime/syscall.Syscall6;[k] entry_SYSCALL_64_after_hwframe;[k] kretprobe_trampoline;[k] kretprobe_trampoline;[k] trampoline_handler;[k] __kretprobe_trampoline_handler;[k] _raw_spin_unlock_irqrestore 1 click click/curing-client-7c6d4bbfb8-rbkgv client ./client node-01
2025-10-22 16:11:52.406689359 +0000 UTC 10049 runtime.goexit.abi0;main.main.gowrap2;github.com/amitschendel/curing/pkg/client.(*CommandPuller).Run;github.com/amitschendel/curing/pkg/client.(*CommandPuller).connectReadAndProcess;github.com/amitschendel/curing/pkg/client.(*CommandPuller).processCommands;github.com/amitschendel/curing/pkg/client.(*CommandPuller).sendGobRequest;encoding/gob.(*Encoder).Encode;encoding/gob.(*Encoder).EncodeValue;encoding/gob.(*Encoder).sendTypeDescriptor;encoding/gob.(*Encoder).sendType;encoding/gob.(*Encoder).sendActualType;encoding/gob.(*Encoder).sendType;encoding/gob.(*Encoder).sendActualType;encoding/gob.(*Encoder).writeMessage;github.com/amitschendel/curing/pkg/client.(*NetworkRWer).Write;net.(*conn).Write;net.(*netFD).Write;internal/poll.(*FD).Write;syscall.write;syscall.Syscall;internal/runtime/syscall.Syscall6;[k] entry_SYSCALL_64_after_hwframe;[k] do_syscall_64;[k] syscall_enter_from_user_mode;[k] syscall_trace_enter.constprop.0;[k] __traceiter_sys_enter;[k] __bpf_trace_sys_enter;[k] bpf_trace_run2;[k] bpf_prog_0f882ec0ca7e929e_ig_seccomp_e;[k] __htab_map_lookup_elem;[k] lookup_nulls_elem_raw 1 click click/curing-client-7c6d4bbfb8-rbkgv client ./client node-01
2025-10-22 16:11:52.406689359 +0000 UTC 10085 runtime.goexit.abi0;net.(*Resolver).goLookupIPCNAMEOrder.func3.gowrap1;net.(*Resolver).goLookupIPCNAMEOrder.func3.1;net.(*Resolver).tryOneName;net.(*Resolver).exchange;net.dnsPacketRoundTrip;net.(*conn).Write;net.(*netFD).Write;internal/poll.(*FD).Write;syscall.write;syscall.Syscall;internal/runtime/syscall.Syscall6;[k] entry_SYSCALL_64_after_hwframe;[k] kretprobe_trampoline;[k] __x64_sys_write;[k] ksys_write;[k] vfs_write;[k] new_sync_write;[k] sock_write_iter;[k] __sock_sendmsg;[k] inet_sendmsg;[k] udp_sendmsg;[k] udp_send_skb;[k] ip_send_skb;[k] ip_output;[k] ip_finish_output;[k] __ip_finish_output;[k] ip_finish_output2;[k] __local_bh_enable_ip;[k] do_softirq;[k] do_softirq_own_stack;[k] asm_call_sysvec_on_stack;[k] __softirqentry_text_start;[k] net_rx_action;[k] process_backlog;[k] __netif_receive_skb;[k] __netif_receive_skb_one_core;[k] __netif_receive_skb_core.constprop.0;[k] br_handle_frame;[k] br_nf_pre_routing;[k] setup_pre_routing;[k] sock_wfree;[k] sock_def_write_space;[k] __wake_up_sync_key;[k] __wake_up_common_lock;[k] _raw_spin_unlock_irqrestore 1 click click/curing-client-7c6d4bbfb8-rbkgv client ./client node-01
2025-10-22 16:13:22.516908876 +0000 UTC 10400 runtime.goexit.abi0;github.com/iceber/iouring-go.New.gowrap1;github.com/iceber/iouring-go.(*IOURing).run;github.com/iceber/iouring-go.(*IOURing).getCQEvent;runtime.selectgo;runtime.gopark;runtime.mcall;runtime.park_m;runtime.schedule;runtime.findRunnable;runtime.stealWork;runtime.(*timers).check;runtime.nanotime1.abi0;[m] [vdso] + 0x7ffd5a3e6a09;[k] entry_SYSCALL_64_after_hwframe;[k] do_syscall_64;[k] syscall_enter_from_user_mode;[k] syscall_trace_enter.constprop.0;[k] __traceiter_sys_enter;[k] __bpf_trace_sys_enter;[k] bpf_trace_run2 1 click click/curing-client-7c6d4bbfb8-rbkgv client ./client node-01
2025-10-22 16:14:22.570940379 +0000 UTC 10674 runtime.goexit.abi0;net.(*Resolver).goLookupIPCNAMEOrder.func3.gowrap1;net.(*Resolver).goLookupIPCNAMEOrder.func3.1;net.(*Resolver).tryOneName;net.(*Resolver).exchange;net.dnsPacketRoundTrip;net.(*conn).Write;net.(*netFD).Write;internal/poll.(*FD).Write;syscall.write;syscall.Syscall;internal/runtime/syscall.Syscall6;[k] entry_SYSCALL_64_after_hwframe;[k] do_syscall_64;[k] syscall_exit_to_user_mode;[k] __traceiter_sys_exit;[k] __bpf_trace_sys_exit;[k] bpf_trace_run2;[k] bpf_prog_a4331b93d3bc1065_ig_cap_sys_exit;[k] htab_map_delete_elem 1 click click/curing-client-7c6d4bbfb8-rbkgv client ./client node-01
Stacktrace Explanation
Last Step: we real time compile a bpftrace and find what this rootkit has been reading
mkdir dfsnoop
cat <<EOF > dfsnoop/vis.json
{
"variables": [
{
"name": "comm",
"type": "PX_STRING",
"description": "The process command to filter the dc data based on",
"defaultValue": ""
}
],
"globalFuncs": [
{
"outputName": "dc_snoop",
"func": {
"name": "dc_snoop",
"args": [
{
"name": "comm",
"variable": "comm"
}
]
}
}
],
"widgets": []
}
EOF
cat <<'EOF' > dfsnoop/dfsnoop.pxl
import pxtrace
import px
program = """
#include <linux/fs.h>
// from fs/namei.c:
struct nameidata {
struct path path;
struct qstr last;
// [...]
};
// comment out this block to avoid showing hits:
kprobe:lookup_fast,
kprobe:lookup_fast.constprop.*
{
$nd = (struct nameidata *)arg0;
printf(\"time_:%llu pid:%d comm:%s t:%s file:%s\",
nsecs, pid, comm, \"R\", str($nd->last.name));
}
kprobe:d_lookup
{
$name = (struct qstr *)arg1;
@fname[tid] = $name->name;
}
kretprobe:d_lookup
/@fname[tid]/
{
printf(\"time_:%llu pid:%d comm:%s t:%s file:%s\",
nsecs, pid, comm, \"M\", str(@fname[tid]));
delete(@fname[tid]);
}
"""
def dc_snoop(comm: str):
table_name = 'dc_snoop'
pxtrace.UpsertTracepoint('dc_snoop',
table_name,
program,
pxtrace.kprobe(),
"10m")
df = px.DataFrame(table=table_name)
df = df[px.contains(df.comm, comm)]
return df
EOF
px run -f ./dfsnoop -- --comm ${COMM}
We real time compile a bpftrace and find what this rootkit has been reading
laborant@dev-machine:honeycluster$ px run -f dcsnoop.px
Pixie CLI
*******************************
* ENV VARS
* PX_CLOUD_ADDR=getcosmic.ai
*******************************
Table ID: output
TIME PID COMM T FILE
2025-11-09 22:27:59.365371839 +0000 UTC 20593 client R etc/resolv.conf
2025-11-09 22:27:59.36538673 +0000 UTC 20593 client R resolv.conf
2025-11-09 22:27:59.365412363 +0000 UTC 20593 client R etc/nsswitch.conf
2025-11-09 22:27:59.365418946 +0000 UTC 20593 client R nsswitch.conf
2025-11-09 22:27:59.365601866 +0000 UTC 20593 client R etc/hosts
2025-11-09 22:27:59.36560909 +0000 UTC 20593 client R hosts
2025-11-09 22:27:59.376590723 +0000 UTC 20593 client R var/run/secrets/kubernetes.io/serviceaccount/token
2025-11-09 22:27:59.376602736 +0000 UTC 20593 client R run/secrets/kubernetes.io/serviceaccount/token
2025-11-09 22:27:59.376608569 +0000 UTC 20593 client R secrets/kubernetes.io/serviceaccount/token
2025-11-09 22:27:59.376613832 +0000 UTC 20593 client R kubernetes.io/serviceaccount/token
2025-11-09 22:27:59.376620668 +0000 UTC 20593 client R serviceaccount/token
2025-11-09 22:27:59.376627658 +0000 UTC 20593 client R token
2025-11-09 22:27:59.376635041 +0000 UTC 20593 client R ..data/token
2025-11-09 22:27:59.376641473 +0000 UTC 20593 client R ..2025_11_09_22_25_54.25061200
2025-11-09 22:27:59.376647379 +0000 UTC 20593 client R token
2025-11-09 22:27:59.431891593 +0000 UTC 20593 client R etc/shadow
2025-11-09 22:27:59.431899959 +0000 UTC 20593 client R shadow
Level up your Server Side game — Join 20,000 engineers who receive insightful learning materials straight to their inbox
