Aggregate Cluster Logs with Loki and Alloy
Metrics tell you that something is wrong; logs tell you why. In this challenge you add the logs pillar to a Rancher cluster that already has the metrics stack running. You install Grafana Loki to store logs and Grafana Alloy to collect them from every node, then wire Loki into the existing Grafana and prove you can query your cluster's logs with LogQL.
You work from the dev-machine workstation, where kubectl and helm are configured. The monitoring stack (including Grafana) is already installed, exactly as it would be after setting up observability - your job is the logging pipeline on top of it.
Step 1: Install Loki
Install Loki in a small, single-binary shape with filesystem storage so it fits alongside Rancher and the monitoring stack on this node. A single loki pod in a loki namespace is enough.
Hint 1 - the Loki chart
The Loki chart now lives in the grafana-community Helm repository (grafana-community/loki). For a lab, deploy it in single-binary mode with filesystem storage and the other scaling components turned off, so only one Loki pod runs. A values file is the cleanest way to pass all those settings.
Step 2: Ship Logs with Alloy
Install a log collector that runs on every node, discovers pods, and pushes their logs to Loki. Grafana Alloy is the current collector for this (it replaced the end-of-life Promtail).
Hint 2 - Alloy as a DaemonSet
The Alloy chart is grafana/alloy (the classic grafana repo, not grafana-community). Run it as a DaemonSet so one agent lands on each node. Its config needs a Kubernetes pod discovery, a step that relabels pod metadata into log labels, a source that tails the pods' logs, and a write step pointing at Loki's push endpoint: http://loki.loki.svc.cluster.local:3100/loki/api/v1/push.
Step 3: Query Logs in Grafana
Add Loki as a data source in the Grafana that is already running, then confirm a LogQL query returns log lines. Do not stand up a second Grafana - reuse the one from the monitoring stack.
Hint 3 - wire Loki into Grafana
Grafana's admin password is in the monitoring-grafana secret. Add a data source of type Loki with the URL http://loki.loki.svc.cluster.local:3100, then query a broad selector like {namespace="kube-system"} in Explore. If the query returns nothing, give Alloy a moment to ship the first lines, and check that its DaemonSet pods are running.