Identify the Source of Unexpected Requests in a Kubernetes Cluster
The company runs its services in a Kubernetes cluster: every service is a group of identical pods managed by a Deployment, a StatefulSet, or another controller. Pods come and go all the time, and every new pod gets a new IP address.
The inventory service is one of them. It runs as the inventory Deployment in the warehouse namespace, and its regular clients are well known:
the orders Deployment from the shop namespace looks up items through the API,
and the uptime-prober Deployment from the monitoring namespace polls the /health endpoint every few seconds.
Recently, the service started receiving requests from unexpected pod addresses, and nobody in your team can explain where they are coming from.
You're given access to the dev-machine with kubectl configured as a cluster admin.
The inventory service writes an access log to the standard output of its container.
The log records the source IP address of every request.
Find out who is behind the unexpected requests.
The Workloads With an Owner
The company has a simple ownership convention:
every top-level workload object (a Deployment, a StatefulSet, a DaemonSet, etc.) carries a team label with the name of the team that owns it.
The label is set only on the top-level object. The pods and any intermediate objects do not have it.
Some of the unexpected requests come from pods that are managed by such workloads.
There are two such workloads, and they are owned by two different teams.
Identify the teams by the values of the team label, in any order.
Hint 1
Container logs are available through kubectl logs, and it accepts a Deployment reference in place of a pod name.
Every line of the access log starts with the client's IP address, and the regular clients make up most of the lines.
You can use text tools like awk, sort, and uniq to produce the list of distinct addresses with the number of requests per address.
Hint 2
The wide output format of kubectl get pods has an IP column, and the command can list pods from all namespaces at once.
Finding the pods of the orders and uptime-prober Deployments in this list can tell you which source addresses belong to the expected clients.
A Deployment usually runs more than one pod, so expect a workload to stand behind several addresses.
Hint 3
Kubernetes objects point to their controllers.
A pod created by a controller has an ownerReferences entry in its metadata, and kubectl describe pod shows the same information in the Controlled By field.
The owner of a pod is not always the top-level object.
For example, a Deployment manages its pods through an intermediate ReplicaSet, and the ReplicaSet has an owner reference of its own.
Follow the references up until you reach an object that has no owner. This is where the team label lives.
The Pod Without an Owner
One unexpected source is a pod that no controller manages.
Somebody likely started it by hand for a quick experiment and forgot to clean it up.
It has no owner reference and no team label, so there is nobody to ask about it.
What is the name of this pod?
The Address Without a Pod
One more source address does not belong to any pod in the cluster.
The pod that used this address is already gone, and the Kubernetes API keeps no record of deleted pods.
Which address is it?
Hint 4
A pod IP address tells you who the client is only while the pod exists. When a pod gets deleted, its address returns to the pool, and the cluster forgets which pod had it.
The access log may still help to make a guess. Compare the requests from this address (the paths and the user agent) with the requests of the clients that you have already identified.