Observability and Monitoring

You cannot operate a cluster you cannot see. This lesson installs the observability stack that Rancher Monitoring is built on - Prometheus for metrics, Grafana for dashboards, and Alertmanager for alerts - and has you define an alert of your own. You install it with Helm from the dev-machine workstation, the same way you installed Rancher itself.

The stack: the Prometheus operator manages Prometheus (metrics), Grafana (dashboards), and Alertmanager (alert routing).

Rancher Monitoring and What It Packages

Rancher offers a one-click Monitoring app (under Cluster Tools in the UI) that deploys a full metrics-and-alerting stack and wires Prometheus and Grafana into the Rancher dashboard. Under the hood, that app is the community kube-prometheus-stack Helm chart with a Rancher wrapper. In this lesson you install that same chart directly, so you can see exactly what Rancher Monitoring deploys and where its pieces live. Everything you learn here maps straight to the Rancher UI.

Step 1: Install the Monitoring Stack

Add the prometheus-community Helm repository and install kube-prometheus-stack into a monitoring namespace. The --set flags trim the footprint so it fits this lab node - short metric retention and no persistent volumes - and expose Grafana on a NodePort (30300) so you can open it in the browser through the Grafana tab.

From the dev-machine terminal:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install monitoring prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace \
  --set grafana.service.type=NodePort \
  --set grafana.service.nodePort=30300 \
  --set prometheus.prometheusSpec.retention=2h \
  --set prometheus.prometheusSpec.resources.requests.cpu=100m \
  --set prometheus.prometheusSpec.resources.requests.memory=256Mi \
  --set grafana.resources.requests.cpu=50m \
  --set grafana.resources.requests.memory=128Mi \
  --set alertmanager.alertmanagerSpec.resources.requests.cpu=25m \
  --set alertmanager.alertmanagerSpec.resources.requests.memory=64Mi

Step 2: Confirm Prometheus and Grafana Are Running

The stack takes a couple of minutes to pull images and start. Watch it come up:

kubectl -n monitoring get pods

You are looking for a running prometheus-* pod and a grafana pod, alongside the operator, Alertmanager, and the exporters.

Step 3: Open Grafana in the Browser

Because you installed Grafana with grafana.service.type=NodePort on port 30300, this playground's Grafana tab points straight at it. Open the Grafana tab - Grafana loads in a new browser tab and shows its login page.

Log in with the username admin. The password is generated at install and stored in a secret; print it from the dev-machine terminal:

kubectl -n monitoring get secret monitoring-grafana \
  -o jsonpath='{.data.admin-password}' | base64 -d; echo

Copy that value into the Grafana login.

Once you are in, browse Dashboards and open one of the "Kubernetes / Compute Resources" views to see live cluster metrics. Because this cluster already runs Rancher, the graphs are full of real activity from the moment you open them. This is the same Grafana that Rancher Monitoring surfaces inside the Rancher UI.

Grafana's Dashboards list, pre-populated by kube-prometheus-stack - the "Kubernetes / Compute Resources" views show live cluster metrics.

Step 4: Break Something, Then Alert on It

Metrics are only half of observability - you also want to be told when something is wrong. The best way to see alerting work is to cause a real problem and watch Prometheus catch it. Deploy a pod that crashes on start, so it enters a crash loop:

kubectl create namespace demo
kubectl -n demo run crasher --image=busybox --restart=Always -- /bin/sh -c "exit 1"

The container exits immediately, Kubernetes restarts it, it exits again, and within a minute it is in CrashLoopBackOff with a climbing restart count:

kubectl -n demo get pod crasher

Now define a PrometheusRule that alerts when that pod has restarted too many times. Prometheus fires alerts from these objects, and the operator only adopts rules whose labels match its selector - so include the stack's release label:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: my-alerts
  namespace: monitoring
  labels:
    release: monitoring
spec:
  groups:
    - name: example
      rules:
        - alert: PodCrashLooping
          expr: kube_pod_container_status_restarts_total{namespace="demo", pod=~"crasher.*"} > 3
          for: 1m
          labels:
            severity: warning
          annotations:
            summary: "The crasher pod is crash-looping"

Apply it from the workstation:

kubectl apply -f my-alerts.yaml

Applying the PrometheusRule - the cluster confirms prometheusrule.monitoring.coreos.com/my-alerts created.

vi my-alerts.yaml

cat <<'EOF' > my-alerts.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: my-alerts
  namespace: monitoring
  labels:
    release: monitoring
spec:
  groups:
    - name: example
      rules:
        - alert: PodCrashLooping
          expr: kube_pod_container_status_restarts_total{namespace="demo", pod=~"crasher.*"} > 3
          for: 1m
          labels:
            severity: warning
          annotations:
            summary: "The crasher pod is crash-looping"
EOF

A monotonic counter stays above the threshold once crossed (left), while a sliding-window increase can dip and reset the alert timer (right).

Step 5: Watch the Alert Fire

An alert does not fire the instant you create it. It moves through states: inactive (condition false), pending (condition true, waiting out the for: window), then firing. With a crash-looping pod the whole trip takes a few minutes - crash-loop backoff slows the restarts, then Prometheus scrapes, then the one-minute for: elapses.

An alert travels from inactive to pending to firing; once firing, Prometheus hands it to Alertmanager for routing.

Watch the state change from the dev-machine terminal. This loop briefly forwards the Prometheus port to the workstation and queries the alert's state every 15 seconds:

while true; do
  kubectl -n monitoring port-forward svc/monitoring-kube-prometheus-prometheus 9099:9090 >/dev/null 2>&1 &
  PF=$!; sleep 3
  STATE=$(curl -sf -G 'http://127.0.0.1:9099/api/v1/query' \
    --data-urlencode 'query=ALERTS{alertname="PodCrashLooping"}' \
    | grep -o '"alertstate":"[a-z]*"' | head -1)
  kill $PF 2>/dev/null
  echo "$(date +%T) PodCrashLooping: ${STATE:-inactive}"
  echo "$STATE" | grep -q firing && break
  sleep 12
done

You will see it print pending for a minute or so, then flip to firing. Press Ctrl+C to stop the loop once it fires.

You can also see it visually in Grafana. Open the Grafana tab, go to Alerting > Alert rules, and find PodCrashLooping - it shows the same Pending, then Firing transition. Alertmanager (which the stack also installed) is what would route that firing alert to email, Slack, or PagerDuty in a real setup.

The operator only loads PrometheusRules whose labels match its selector; a rule missing the release label is silently ignored.

Step 6: Build Your Own Dashboard

Browsing the built-in dashboards is one thing; building your own is where Grafana earns its place. Now that the crasher pod has been restarting for a few minutes, its restart count is a perfect live metric to graph. You will create a small dashboard with a single panel that tracks it.

kubectl -n monitoring get secret monitoring-grafana \
  -o jsonpath='{.data.admin-password}' | base64 -d; echo

Open the Grafana tab and, in the left menu, choose Dashboards > New > New dashboard. Grafana asks how to lay out the dashboard - choose Auto grid (the simpler default; with a single panel the choice does not matter).

A new dashboard offers an Auto grid or Custom grid layout - Auto grid is fine here.

Now add a panel: click the + Add new element button and choose Panel. The empty panel appears with a blue Configure visualization button in its middle - click it to open the panel editor.

A new panel shows a blue "Configure visualization" button in the middle - click it to open the panel editor.

The data source should already be Prometheus (the stack wired it in as the default). In the Query section below the graph, switch the query mode from Builder to Code - notice how the UI changes to a free-text field reading Enter a PromQL query. Type:

kube_pod_container_status_restarts_total{namespace="demo"}

Switch the Query section from Builder to Code to type raw PromQL in the "Enter a PromQL query" field.

Give the panel a title like "Crasher restarts", then Save the dashboard with any name you like (for example, "My Cluster Health"). The panel immediately shows the crasher's climbing restart count - the same metric your alert rule watches, now visualized.

The finished panel graphing the crasher's restart count - built from a single PromQL query.

You're Done

You installed the Prometheus and Grafana stack that underpins Rancher Monitoring, browsed the built-in dashboards, caused a real failure, watched your own alert catch it and fire, and built a dashboard panel from a raw metric. That is the core observability loop: collect metrics, visualize them, and alert on the conditions that matter. On a production cluster you would enable Rancher's Monitoring app for the same stack with UI integration, but the objects you worked with here - Prometheus, Grafana, and PrometheusRule - are identical.

This lesson covered the metrics pillar of observability. Logs and traces are the other two; the next lesson adds the logs pillar by installing Grafana Loki and the Alloy log shipper, then querying your cluster's logs in this same Grafana.

The challenge below has you stand up the stack, break a pod, and prove your alert fires. Solving it records your progress.