Kubernetes Observability On-Call: Six Tickets with Prometheus, Loki, and Jaeger

You're on call today at HotROD, a small ride-hailing company. Six support tickets are waiting for you. Each one is a real problem in the cluster in front of you. To solve them, you'll use metrics, logs and traces, and part of the job is choosing the right one.

This lab tests what you know. It doesn't teach the theory. If a term is new to you, read the companion post, How observability works in Kubernetes. Each ticket links to the section you need.

How a ticket works

1. Read the message from the person who opened the ticket.
2. Do the steps under See the problem. They tell you where to look and what you'll see while the problem is still there. Seeing it first is part of the job, and it gives you something to compare with later.
3. Read the Done when line. It says exactly what the lab checks, and what you'll see once it's fixed.
4. Investigate and fix the problem, any way you like.
5. The lab checks the cluster every few seconds and closes the ticket by itself when the problem is fixed.

Two tickets ask a question instead. Type your answer and click Submit. If the box turns red, that isn't the answer yet, so look again and try another.

Tickets unlock one at a time, in order. If you get stuck, open the Nudge first, and the Solution only if you need it.

Where things run

Every command and query in this lab says where it goes:

• Terminal: the dev-machine tab. Commands in sh blocks, such as kubectl, run here.
• Prometheus tab, Query page: PromQL queries. Paste the query into the box at the top, click Execute, and read the Table view.
• Grafana tab, Explore: LogQL queries for logs. Pick the Loki data source and switch the editor to Code.
• Jaeger tab: traces. There are no queries to type; you search and click.

The System You're On Call For

HotROD, the main product

HotROD ("Rides on Demand") is the company's app. A customer asks for a ride, and HotROD finds the driver who can get there first. It's Jaeger's demo application: four small services that run together in one Deployment, hotrod, in the demo namespace.

Open the HotROD tab and click a customer to book a ride yourself.

Every ride request follows the same path:

 customer ──▶ frontend  GET /dispatch?customer=123
                │
                ├─1─▶ customer   GET /customer       "where is this customer?"
                │        └─▶ MySQL   SQL SELECT
                │
                ├─2─▶ driver     FindNearest (gRPC)  "which drivers are close?"
                │        └─▶ Redis   FindDriverIDs, then GetDriver for each one
                │
                ├─3─▶ route      GET /route          "how long would this driver take?"
                │                (one call per candidate driver)
                │
                └──▶ answer: the driver with the shortest ETA

A few things to know about HotROD:

• MySQL and Redis aren't real databases. HotROD simulates them. They appear in traces as the services mysql and redis-manual, but there are no Pods for them.
• HotROD produces all three signals: Prometheus metrics, structured logs that include a trace ID, and a trace for every request.
• A Deployment called loadgen books a ride every 2 seconds, so there's always traffic to look at.

The other services

Three smaller services run next to HotROD in demo. They're stand-ins: each does just enough to act like the real thing.

The observability stack

Everything in the observability namespace is there to watch the services above:

Before You Start: Your Tools

You'll switch between these tabs. Take a minute to find your way around them before the first ticket.

Ticket 1: "We can't see checkout's metrics"

From: Priya, lead developer of checkout

We deployed checkout to demo and a colleague added a ServiceMonitor for it. It's been an hour and there's nothing in Prometheus. The app serves metrics on /metrics. Can you get it scraped?

See the problem. In the Prometheus tab, Query page, run:

up{namespace="demo", service="checkout"}

You'll see Empty query result. up is the series Prometheus records for every target it scrapes, so an empty result means Prometheus isn't scraping checkout at all. Under Status > Target health, there's no checkout target either.

Done when: Prometheus scrapes checkout. Run the same query again: it returns one row with the value 1. A new target can take a minute or two to appear.

Theory: how Prometheus finds what to scrape.

kubectl -n demo get servicemonitor checkout -o yaml
kubectl -n demo get service checkout -o yaml

kubectl -n demo patch servicemonitor checkout --type=json -p '[
  {"op": "replace", "path": "/spec/selector/matchLabels", "value": {"app.kubernetes.io/name": "checkout"}},
  {"op": "replace", "path": "/spec/endpoints/0/port", "value": "http"}
]'

Ticket 2: "Checkout was down and nobody got paged"

From: Sam, engineering manager (after this morning's incident review)

Last night a bad rollout scaled checkout to zero for five minutes. The CheckoutDown alert exists, but it never fired. We want it to fire within 2 minutes whenever checkout has no healthy target - including when there are no pods at all.

See the problem.

1. In the Prometheus tab, open the Alerts page. CheckoutDown is listed as inactive: its condition isn't true, so it isn't about to fire. Expand it to see its expression and its for time.
2. In the terminal, look at the rule as a Kubernetes object:

   kubectl -n demo get prometheusrule checkout-alerts -o yaml
   

   Copy to clipboard
3. In the Prometheus tab, Query page, run the rule's expression, up{job="checkout-svc"} == 0. You'll see Empty query result, and it stays empty whether checkout is up or down. Then run up{job="checkout"} and compare.

Done when: all three of these are true:

1. you've fixed the CheckoutDown alert rule,
2. you've scaled checkout to 0 replicas to test it, and
3. CheckoutDown is firing. On the Alerts page it turns pending first, then firing.

You'll bring checkout back at the very end of the lab.

Theory: alerting on symptoms.

kubectl -n demo patch prometheusrule checkout-alerts --type=json -p '[
  {"op": "replace", "path": "/spec/groups/0/rules/0/expr", "value": "absent(up{job=\"checkout\"} == 1)"},
  {"op": "replace", "path": "/spec/groups/0/rules/0/for", "value": "1m"}
]'
kubectl -n demo scale deployment checkout --replicas=0

Ticket 3: "Rides got slower after this morning's change"

From: Lena, HotROD backend team

Someone changed HotROD's configuration this morning. Since then, booking a ride takes about a second instead of 0.7 s. Nothing is failing and the logs look normal. Which service is now the bottleneck?

See the problem. In the Jaeger tab, choose the service frontend and click Find Traces. Each frontend: GET /dispatch trace takes about 1 second. Click one to open its timeline: every bar is a span, and its position shows when it started.

Done when: you type the name of the service that is causing the extra time into the answer box below, and the lab accepts it.

Theory: reading a trace.

Ticket 4: "...and fix it, please"

From: Lena, HotROD backend team

Thanks! Can you undo whatever causes it?

See the problem. In the Prometheus tab, Query page, run this query. It's the average time HotROD took to answer a ride request over the last minute, in seconds:

sum(rate(hotrod_request_latency_sum{endpoint="GET_/dispatch"}[1m]))
  / sum(rate(hotrod_request_latency_count{endpoint="GET_/dispatch"}[1m]))

Right now it's about 1 (one second).

Done when: both of these are true:

1. HotROD no longer runs with the setting that slowed it down, and
2. the same query returns less than 0.85. Give it a minute after your fix, because it's an average over the last minute.

kubectl -n demo get deploy hotrod -o jsonpath='{.spec.template.spec.containers[0].args}'

kubectl -n demo patch deploy hotrod --type=json \
  -p '[{"op": "replace", "path": "/spec/template/spec/containers/0/args", "value": ["all"]}]'

Ticket 5: "Who keeps getting declined?"

From: Marco, customer support

api-gateway logs every payment as JSON. We want to call the customer with the most rejected payments in the last 15 minutes.

See the problem. In the Grafana tab, open Explore, pick the Loki data source, switch the editor to Code, and run:

{app="api-gateway"}

You'll see a stream of JSON lines, one per payment, with fields such as level, msg and customer. There are far too many to count by eye.

Done when: you type that customer's ID into the answer box below, and the lab accepts it.

Theory: labels versus log content.

topk(3, sum by (customer) (count_over_time({app="api-gateway"} | json | level="error" [15m])))

Ticket 6: "image-resizer is slow, but nothing's wrong with it"

From: Aisha, driver onboarding team (owners of image-resizer)

Resizing an image takes about 7 seconds. On a laptop it takes one. The Pod is Running, it never restarts, and the logs are clean.

See the problem. In the terminal, read the last few log lines:

kubectl -n demo logs deploy/image-resizer --tail=5

Each line shows how long one image took, in the seconds field. Right now it's about 7.

Done when: the same command shows seconds of 2 or less on the last three lines, from a Pod that's running your fix.

Theory: saturation and the USE method.

rate(container_cpu_usage_seconds_total{namespace="demo", container="resizer"}[1m])

kubectl -n demo set resources deployment image-resizer -c resizer --limits=cpu=1
kubectl -n demo logs deploy/image-resizer -f    # "seconds" drops to 0 or 1

Close out

See the problem. checkout is still scaled to 0 from Ticket 2, so on the Prometheus tab's Alerts page, CheckoutDown is still firing.

Done when: all three of these are true:

1. checkout is running again,
2. up{namespace="demo", service="checkout"} returns 1 on the Prometheus tab's Query page, and
3. CheckoutDown is back to inactive on the Alerts page, by itself.

kubectl -n demo scale deployment checkout --replicas=1

Shift Report

Want more? Run o11y-lab mystery in the terminal for a random incident with no ticket at all.