Grafana Alerts Lab Playground

Trigger the alert

1. Open Grafana on the left and Alert inbox on the right. The dashboard shows five services spread across the VMs.
2. Click Trigger checkout fault in Alert inbox. Checkout's error ratio rises to 75%, Grafana fires its provisioned alert, and a popup appears in the inbox.
3. Click Restore checkout to receive the resolved notification.

Grafana login for exploring the alert rule and contact point: admin / lab-demo.

Create your own alert and webhook in Grafana

1. Sign in to the Grafana tab as admin / lab-demo. Open Alerts & IRM → Alerting → Notification configuration → Contact points, click New contact point, choose Webhook, enter your URL, and save.
2. Open Alerts & IRM → Alert rules → New alert rule. Query Prometheus with lab_service_error_ratio{service="checkout"} and set the condition to is above 0.2.
3. Choose the Lab folder and create a new evaluation group, such as my-alerts, with a 10 second interval. Set the pending period to None or 0s.
4. Under notifications, choose Select contact point and select your new webhook contact point. Save the rule, then click Trigger checkout fault in Alert inbox to test it.

The supplied checkout rule is file-provisioned and read-only in Grafana. Your new rule can send directly to your webhook; it does not need a notification policy change. Both checkout rules may notify you for the same fault. The Lab alerts contact point and default policy are editable in the UI.

Send the alert to your own destination

Grafana's Lab alerts contact point always notifies Alert inbox. Under Your destinations in Alert inbox, you can add one webhook URL and one PagerDuty integration key, and Grafana notifies them too. Grafana picks up a saved destination within about 10 seconds.

Add your destination before you click Trigger checkout fault. An alert that is already firing is not resent to a replacement webhook URL until it resolves.

Your phone, with ntfy

1. Install the ntfy app and subscribe to a hard to guess topic name, such as grafana-lab-7f3k9q. Anyone who knows the topic name can read its messages.
2. Paste https://ntfy.sh/<your-topic> into Webhook URL and click Save webhook. Alert inbox adds ntfy's template=grafana option, so the push notification reads as an alert instead of raw JSON.

Discord

1. In a Discord server where you can manage webhooks, open the channel's Edit Channel → Integrations → Webhooks, create a webhook, and click Copy Webhook URL.
2. Paste it into Webhook URL and click Save webhook. Grafana sends it through its Discord integration.

Slack incoming webhook URLs (https://hooks.slack.com/...) work the same way. Any other URL, such as one from webhook.site, receives Grafana's webhook JSON unchanged.

PagerDuty

1. On a PagerDuty service, add an Events API v2 integration and copy its 32 character Integration Key.
2. Paste it into PagerDuty integration key and click Save PagerDuty key. A firing alert opens an incident and the resolved notification closes it.

Click Remove to stop sending to a destination. Destinations last for this session only. Grafana masks Discord, Slack, and PagerDuty secrets, but anyone who can open this session's Grafana can read a plain webhook URL such as an ntfy topic. Links inside the notifications point to Grafana's address inside the VM (localhost:3000), so open Grafana from its tab instead.