Events, the first line of diagnosis

Before installing Prometheus, before setting up Grafana, before arguing about traces: Kubernetes is already telling you what is going on, and it does so in a place almost nobody reads carefully.

The book's Observability chapter walks through metrics, logs and traces, the three signals you have to instrument. This module starts earlier: with what the cluster is already telling you for free.

An Event is an API object, like a Pod or a ConfigMap. A cluster component (the scheduler, the kubelet, a controller) emits one when something worth mentioning happens to another object. kubectl describe shows them at the end, and that "at the end" is symbolic: it is where most people stop reading, right before the answer.

In this lesson there are three broken Pods waiting for you. Each one fails for a different reason, and each one has a different component complaining about it.

kubectl get pods

Step 1: imagen-fantasma, the kubelet's failure

kubectl describe pod imagen-fantasma

Scroll down to Events. You will see a sequence: the node assigned, an attempt to pull the image, and a Warning with the failure.

That Warning spells out that the manifest for that tag does not exist in the registry. There is nothing to guess: the error comes from the container registry, goes through the runtime, and the kubelet publishes it in the API for you to read.

Notice the event's Age too: something like 2m (x5 over 3m). That means the same event has repeated five times and Kubernetes has grouped it into a single line with a counter, instead of flooding you with five identical entries.

Step 2: nodo-imposible, the scheduler's failure

This Pod has not even reached a node. It asks for 500 CPUs.

kubectl events --for pod/nodo-imposible

The Reason tells you it could not be scheduled, and the message details why each node failed: how many for insufficient CPU, how many for taints, how many for affinity. It is, literally, the report of the scheduler's filtering phases you studied in the Scheduling module.

And there is a detail worth your attention: look at who emits this event. It is not the kubelet of any node, because no kubelet has ever seen this Pod. It is the default-scheduler.

That is the real value of Events: they tell you which cluster component has the problem. A FailedScheduling is a capacity or placement-rules problem. A Failed while pulling an image is a node, registry or credentials problem. They look nothing alike, and you diagnose them in different places.

Step 3: api-rota, the failure that repeats

kubectl events --for pod/api-rota

kubectl get pod api-rota

This Pod starts fine. The image is pulled, the container runs... and a few seconds later the kubelet kills it. And again. And again. The RESTARTS column climbs on its own.

The Events tell the whole story: the liveness probe points at a path that returns 404, the kubelet marks it Unhealthy, kills it, and restartPolicy: Always brings it back up. After a while you will also see the BackOff: Kubernetes waiting longer and longer between retries.

Here is the lesson that separates those who know how to diagnose from those who do not:

• kubectl logs api-rota shows you the application (and nginx, knowing nothing about any of this, will say everything is fine).
• kubectl events --for pod/api-rota shows you Kubernetes talking about the application.

The container is not dying. It is being killed, and the reason does not appear anywhere in the application logs. Ever. If you only look at the logs, this bug can cost you an afternoon.

Step 4: Events are objects, and you filter them as such

An Event is an API object with structured fields. That means you can filter, sort and query it like anything else:

# Everything that has happened in the cluster, in order
kubectl events -A

# Only the problems
kubectl events --types=Warning -A

# Everything about one specific object
kubectl events --for pod/api-rota

# And the classic, in case your cluster is old
kubectl get events --field-selector type=Warning,reason=FailedScheduling

Save the cluster's list of warnings, which is the first thing you would do in a real incident:

kubectl events --types=Warning -A > /home/laborant/warnings.txt
cat /home/laborant/warnings.txt

There you have, on a single screen, the three failures of the scenario. Without installing anything.

Step 5: Fix the Pod

You know exactly what is wrong with imagen-fantasma. The image tag does not exist, and a Pod's image cannot be changed in place:

kubectl delete pod imagen-fantasma
kubectl run imagen-fantasma --image=ghcr.io/iximiuz/labs/nginx:alpine

Step 6: The fine print

Now, the part you need to learn before trusting Events.

Events expire. The API server deletes them once the --event-ttl has passed, and by default that is one hour. They are stored in etcd, and etcd is not a logging system: if it kept every event of a large cluster forever, they would eat it alive.

The consequences are very practical and you need to internalize them:

• Last night's incident has no Events this morning. They have evaporated. If nobody collected them at the time, the information no longer exists.
• Events are not an audit trail. That is what the API server's audit log is for, and it is a completely different thing: it records who made which request, and it is kept on disk.
• That is why you export them. Tools like kubernetes-event-exporter, the OpenTelemetry events receiver, or the agents of any observability suite do exactly one thing: they watch the Events and ship them somewhere they last longer than an hour.

In other words: the best argument for setting up the stack of the next lesson is not "I want pretty graphs". It is that Kubernetes throws its best information in the trash every sixty minutes.

The Reasons you will see for the rest of your life

Summary

• Events are API objects emitted by the cluster's components. They are the first line of diagnosis, and they are free.
• The Reason tells you what failed. The emitter tells you who has the problem, and therefore where to keep looking.
• kubectl logs ≠ kubectl events: logs are the application talking; events are Kubernetes talking about the application. A container killed by a broken liveness probe leaves no trace in its own logs.
• kubectl events --types=Warning -A is the command to start any incident with.
• Events expire after an hour and they are not an audit trail. If you care about them, you have to export them. That is the starting point of the next lesson.