Lesson  in  Observability 101: Metrics

Metric Samples

Numeric measurements repeated over time

Overview

Imagine you own a coffee shop. To serve customers well, grow the business, and fix problems quickly, you need more than a feeling for how the shop is doing.

You need facts.

In this lesson, you'll learn what metrics are and how they help you answer business and technical questions.

Objectives:

  • Turn a question into something you can measure
  • Identify the three parts of a sample: metric name, value, and timestamp
  • Explain what one sample says and what repeated samples reveal
  • Tell the difference between an event record (an order) and a metric sample

From Questions to Measurements

There are dozens of things you could observe in a coffee shop, and many of them can be turned into numbers. But which numbers actually help you understand what's going on?

A number is useful when it helps you answer a question or make a decision.

Suppose you want to plan your coffee bean orders: when to order, how much, and which kind. For that, you need to know how demand changes over a day, a week, or a month.

So the first question you could ask is:

How do coffee sales change over a day, a week, or a month?

Once the question is clear, you can pick an observable quantity: something you can count or measure to answer it. For this question, the observable quantity is coffees sold.

Note

💡 Let's pretend for a moment that every type of coffee needs the same amount of beans, so the amount of beans you need is proportional to the number of coffees sold.

Why not measure the remaining coffee beans?

That number tells a similar story, but it's harder to read. The amount of beans on the shelf goes down with every coffee and jumps back up with every delivery. To work out how many coffees you sold, you'd have to subtract every delivery first.

The number of coffees sold only ever goes up. The difference between any two readings is exactly the number of coffees sold in between, no bookkeeping required.

Recording coffee sales over time reveals daily, weekly, and monthly patterns. You can use those patterns to estimate future demand and order beans based on evidence instead of guesswork.

Coffee sales plotted over a day, a week, and a month

Coffee sales plotted over a day, a week, and a month

Now imagine you had started by writing down random things about the shop instead:

  • the number of customers in the shop
  • water consumption
  • noise level
  • anything else you can think of

Weeks later, you ask the bean question and none of that data helps. You can't answer the question, and you spent a lot of time writing down numbers you don't need.

As a rule of thumb: start with the question, not the data.

Note

💡 In real life, this is less of a problem: monitoring systems collect many measurements by default, without any work on your part. As you'll see later in the course, storing metrics is relatively cheap, and those extra numbers may come in handy one day.

What matters is that you collect the right information for the questions you actually have.

Your First Metric

It's time to start collecting data!

You don't need fancy equipment or a monitoring system to make observations: pen and paper are a perfect starting point.

You already decided what to measure: coffee sales. This is going to be your first metric.

A metric is a named measurement:

  • the name says what is being measured
  • each recorded value says how much

Monitoring systems have a few conventions for metric names, and this course follows them from the start so they look familiar later on. Here is your metric's name:

coffees_sold_total
  • Lowercase words, joined with underscores.
  • The name says what is measured (coffees sold), not how or where.
  • The _total suffix marks a running total: a number that only goes up as more coffees are sold.
Why _total and not _count?

In everyday language the two words mean roughly the same thing, but monitoring systems reserve _count for something else: the number of measurements taken, which you'll meet later in the course.

Running totals end in _total. Sticking to the convention keeps the two apart.

Note

💡 Don't worry about naming yet: you'll learn how to design clear metric names later in the course.

Here is the plan: every now and then, you look at the cash register and write down how many coffees have been sold since the shop opened.

Let's do that now: start the playground (if you haven't already). Once it's running, you should see your virtual coffee shop: the Observable Cafe.

The cafe has two panels: the menu, one card per coffee type, and your notebook, where your measurements go.

Clicking on a coffee sells one. Go ahead and make a few sales.

Click the Observe now button.

Congratulations! You have recorded your first observation.

A new line appeared in the notebook: the time, and Coffees sold: followed by your count. That's the human way to write it down. A monitoring system records the very same observation like this:

A sample

Same information, stricter format. It has three parts:

In your notebookIn a monitoring systemMeaning
The time09:00Timestamp: when the observation was made
"Coffees sold"coffees_sold_totalMetric name: what was measured
Your count3Value: how many coffees were sold since opening

(Your time and count will differ, of course.)

One recorded value of a metric at one point in time is called a sample.

Note

💡 Monitoring systems such as Prometheus and VictoriaMetrics call this a sample. OpenTelemetry, a standard for collecting telemetry, calls the same thing a data point.

This course uses sample.

A single sample isn't very useful on its own: it tells you what the value was at 09:00, and nothing more.

To learn anything from it, you need more samples.

Samples Over Time

To spot a trend, you need to see how a metric changes over time, and for that you need more than one sample. How many samples you have matters less than how often you collect them.

The time between two observations is your call. It depends on what you want to find out:

  • The busiest day of the week: one sample at the end of each day
  • Rush hours: one sample at the end of each hour
Note

💡 Different monitoring systems have different names for this. Prometheus and VictoriaMetrics call it the scrape interval when they collect samples by periodically asking (scraping) the application, much like you check the register. OpenTelemetry calls it the export interval.

Whatever the name, it sets the resolution of your data: how often a sample is taken. Sample every hour and you'll see the lunch rush, sample once a day and you won't. More on that later.

Let's say you recorded a sample every hour:

Timecoffees_sold_totalChange from previous sample
08:000-
09:003+3
10:0032+29
11:0035+3
12:0040+5
13:0090+50
14:00120+30
15:00130+10
16:00131+1
17:00131+0

The last column is where the trends show up:

  • Early morning: a slow start
  • Around 10:00: a small morning peak
  • Late morning: quiet
  • Lunch: the rush hour
  • After 15:00: almost nothing

One day isn't enough to draw conclusions. But if the same pattern shows up day after day, you can act on it.

For example: nobody bought anything after 16:00, so you could close an hour earlier.

Or open an hour later and get an extra hour of sleep. 😉

Orders Are Not Samples

Your cash register records every order with all of its details:

  • when the order was placed
  • what drink was ordered
  • how much was charged
  • and so on

A metric sample keeps none of that. All you know is the total number of coffees sold at a given time.

This is both a strength and a limitation. Metrics are cheap to collect and store precisely because they keep so little. But the individual events are gone: you can't tell which order was placed when, or how many drinks one customer bought.

Metrics are great for spotting trends and patterns, and for noticing when something breaks the pattern. They don't answer every question, though: when you need the details of a single event, you need another signal (logs, for example).

Still, because they are cheap and give you a quick overview of the current state, metrics are the first thing worth investing in, whether you run a coffee shop or dozens of applications.

Note

💡 The next lesson shows how metrics can keep some detail about the observed events.

From Shop to Service

The coffee shop is a model. Let's take the same steps with a web service.

Coffee shopWeb service
The questionCoffee sales trendsService traffic trends
The observable quantityCoffees soldRequests handled

Where can you see requests? Most web servers write one line to an access log for every request they handle:

10.0.0.5 - - [26/Aug/2026:09:00:01 +0000] "GET /checkout HTTP/1.1" 200 512
10.0.0.8 - - [26/Aug/2026:09:00:03 +0000] "GET /checkout HTTP/1.1" 200 512
10.0.0.5 - - [26/Aug/2026:09:00:04 +0000] "POST /checkout HTTP/1.1" 500 87

So the simplest observation you can make is counting the lines in that file:

wc -l < /var/log/nginx/access.log

The metric: Name it the same way as the shop's metric: http_requests_total. The _total suffix says it's a running total: requests handled since the log file was created.

The sample: Look at the clock, count the lines, and write both down:

09:00 http_requests_total 3

Samples over time: Count the lines every hour, and you get the same kind of table as in the shop: a slow start, a lunch rush, and a quiet afternoon.

Log lines are not samples. Each access-log line describes one request: who sent it, which page they asked for, and whether it worked. That's the web service's order ticket (event log). The sample keeps only the count.

Side by side:

Coffee shopWeb service
Where events are recordedCash registerAccess log
The metriccoffees_sold_totalhttp_requests_total
A sample09:00 coffees_sold_total 309:00 http_requests_total 3
One eventOne orderOne access-log line
Note

💡 Counting log lines isn't how production services measure traffic.

But it's a real observation you can make with nothing more than a web server and wc -l, and it already tells you something.

If the count is usually around 10,000 by noon and today it's below 1,000, something is wrong. You don't know what yet, but you know to look.

Summary

In this lesson, you learned about the kind of data monitoring systems collect: metrics.

Key takeaways:

  • Measuring starts with a question. The question leads to the observable quantity.
  • A metric is a named measurement. Its name says what is measured, like coffees_sold_total.
  • A sample records one observation: metric name, value, and timestamp. Repeating the observation makes change visible.
  • A sample is not an event record. It keeps the running total the events produced, not the events themselves.
  • The same steps work for a web service: coffees sold becomes requests handled, the cash register becomes the access log, and coffees_sold_total becomes http_requests_total.

Right now, the notebook can only tell you how many coffees were sold. The next lesson adds a way to keep a few more bits of information with each sample.

Previous lesson
How This Course Works
Next lesson
Labels and Series