HotROD + Jaeger on K3s Playground

Overview

Jaeger's HotROD ("Rides on Demand") demo app, pre-deployed on a 3-node K3s cluster and wired to a Jaeger v2 all-in-one backend over OTLP. Nothing to install — the HotROD and Jaeger UI tabs on the right come up live.

Give the cluster a minute after boot. The VMs restore from a snapshot, so K3s and the two pods need ~60–90s to settle; until then the app tabs show an error page. Reload once kubectl -n example-hotrod get pods reports both pods Running.

What's inside

Upstream's examples/hotrod/kubernetes is now just a pointer to examples/oci (Helm + kube-prometheus-stack, built for demo.jaegertracing.io). This playground instead ports the example's docker-compose.yml topology straight to plain manifests — the same two containers, the same wiring, nothing else to learn.

The only wiring that matters is one environment variable on the HotROD pod:

- name: OTEL_EXPORTER_OTLP_ENDPOINT
  value: http://jaeger.example-hotrod.svc.cluster.local:4318

Compose's http://jaeger:4318 becomes the cluster DNS name. That's the whole port.

Take it for a ride

1. Check the deployment. In the dev-machine terminal:

kubectl -n example-hotrod get pods,svc

2. Make a request. Click any of the four customer buttons in the HotROD tab. Each click dispatches a car and produces exactly one trace. From a terminal it's:

curl -s "http://172.16.0.2:30080/dispatch?customer=123" | jq

3. Read the trace. In the Jaeger UI tab, search service frontend, operation GET /dispatch. One dispatch is ~39 spans and ~800 ms, fanning out across six services:

frontend ──► customer ──► mysql          (SQL SELECT, behind a mutex)
         ──► driver   ──► redis-manual   (~13 calls, ~10% fail and retry)
         ──► route                       (×10, limited concurrency)

4. Find the pathologies. The demo has performance bugs baked in on purpose:

• Serialized DB access — SQL SELECT in mysql is guarded by a mutex. Click the customer buttons rapidly and watch this span's duration climb while requests queue behind each other.
• Retried Redis errors — the driver service's FindDriverIDs fails roughly 10% of the time; failed spans show up red in the timeline, followed by a retry.
• Limited fan-out — the ten GET /route calls run only a few at a time, so the timeline shows a stair-step rather than ten parallel bars. frontend is where the concurrency is capped.

5. Pivot to logs. Every HotROD log line carries the trace_id of the request that produced it:

kubectl -n example-hotrod logs deploy/hotrod | grep <trace_id>

6. See the architecture. The System Architecture tab in Jaeger draws the service dependency graph from the traces themselves — nothing declares it. Or check the raw API:

curl -s "http://172.16.0.2:31686/api/dependencies?endTs=$(( $(date +%s) * 1000 ))&lookback=3600000" | jq

Notes

• The "view trace" links in HotROD point at localhost:16686 and won't resolve inside the browser tab — HotROD builds them from a static JAEGER_UI value that can't know the playground's per-run URL. Copy the trace ID into the Jaeger UI tab instead, or point it at the real URL once you know it: kubectl -n example-hotrod set env deploy/hotrod JAEGER_UI=<jaeger tab url>.
• Jaeger's Monitor (SPM) tab won't work here — that needs the Prometheus metrics backend from the upstream Helm stack.
• Storage is in-memory. Restarting the Jaeger pod drops every trace collected so far.
• To redeploy from scratch: kubectl delete ns example-hotrod && kubectl apply -f /home/laborant/hotrod/hotrod.yaml