Lesson  in  Observability 101: Metrics

Metric Types

Counters and gauges, why the type decides which math is valid, and the first measurement that fits neither

Overview

Names and labels identify a time series, but they don't explain how its values should be interpreted: do they add up over time, or do they describe a state at the moment of observation? Without that, you can't read the data correctly.

In this lesson, you'll learn how metric types give a series its meaning, and how that meaning decides which calculations are valid.

Objectives:

  • Choose counters for accumulated events and gauges for current state
  • Interpret counter changes across a reset
  • Decide which calculations preserve a gauge's meaning
  • Select a metric type from the question the stored data must answer
  • Recognize a measurement that fits neither type

Why Metric Types Matter

So far, a metric value has been a number with a name, a label set, and a timestamp. That's enough to store it. It isn't enough to read it.

Take the following samples:

TimeValue
10:4520
11:0024
11:1529

If the metric is coffees_sold_total with drink="latte", the values accumulate:

  • The latest sample is the running total
  • The difference between two samples is what happened in between: nine lattes sold in half an hour
Three samples of the latte series of coffees_sold_total

Each sample holds the previous total plus the lattes sold since then

Note

💡 Adding the samples up gives 73, and the shop did not sell 73 lattes.

The sum is valid arithmetic and a wrong conclusion.

If the same metric is indoor_temperature_celsius, the values describe a state:

  • The difference between 20 and 29 says the room warmed up
  • The average, 24.3 degrees, summarizes the morning
The same three samples drawn as temperature points over time

Each sample is the temperature at that moment, and the average summarizes the period

Note

💡 The sum is still meaningless.

Same values, different valid operations:

OperationCoffees soldTemperature
Latest sampleTotal sold since the count startedCurrent temperature
Difference between two samplesCoffees sold in betweenHow much the room warmed, rarely the question
Average of the samplesMeaninglessTypical temperature over the period
Sum of the samplesMeaninglessMeaningless

What separates the two columns is the metric type.

It answers one question about a series: do the values accumulate, or do they describe a state at the time of observation?

That answer lives in two places:

  • In the definition. When you decide to measure something, you decide how its values behave. The type belongs to your mental model of the metric, next to its name, unit, and labels.
  • In the interpretation. Every calculation over the series has to respect the type. Summing values that accumulate, or subtracting values that describe state, produces a number that answers nothing.

The one place it doesn't live is storage. A stored sample is still a value and a timestamp: 24 looks the same whether it counts lattes or measures degrees. An application can declare the type when it exposes its metrics (makes them available for collection), but the sample itself carries no type, and nothing in storage stops you from doing the wrong math.

So where do you look up the type of a metric you didn't define? In its definition: the declaration the application publishes next to the metric, the documentation, and naming conventions like the _total suffix from the first lesson. You'll see such a declaration in the next module, when the course gets to how metrics are exposed.

Note

💡 This split can feel abstract right now. The next units make it concrete: you'll see where the type shapes how you define and read a metric, and where it quietly disappears from the data.

So the type is decided when the metric is defined, needed whenever the data is read, and absent from the data itself.

Counters

A counter represents a cumulative value. It can count events, such as sales, or accumulate an amount, such as the revenue from today's sales or the grams of coffee beans ground.

It only increases or stays unchanged: between two samples, the value either grew or stayed where it was.

Why write down the total?

Every observation, you copy the running total into the notebook, not the number of sales since you last looked. Here is the latte count over one hour:

Timecoffees_sold_total with drink="latte"What happened since the preceding sample
10:4520First sample
11:00244 sales
11:15295 sales
11:30367 sales
11:45415 sales

The total itself is rarely the question: 41 lattes since opening says nothing about the lunch rush. The answers are in the differences. How many lattes in the last hour? 41 - 20, so 21. Between 11:00 and 11:30? 36 - 24, so 12.

Now suppose you were busy at the counter at 11:15 and never wrote that line. 36 - 24 still says 12. Nothing was lost. Had you been writing down "5 since I last looked" instead of the total, the skipped observation would have taken five sales with it.

A running total forgives a missed sample. That's why counters accumulate.

CalculationResultVerdict
Latest sample41Lattes sold since opening
Difference between two samples: 41 - 2021Lattes sold in the hour
Average of the samples: 150 / 530Meaningless
Sum of the samples150Meaningless

Starting over

Tuesday, 08:30. You want to know how many lattes you sold since this time yesterday, so you flip back a page:

Timecoffees_sold_total with drink="latte"What happened since the preceding sample
Mon 17:1552First sample
Mon 17:30553 sales
Mon 17:45583 sales
Tue 08:152The shop opened with a fresh count, then 2 sales
Tue 08:3064 sales

6 - 52 is -46. The shop did not unsell 46 lattes.

You know exactly what happened: the count is lattes sold since the shop opened, so every morning starts at zero. The drop from 58 to 2 is a counter reset. After it, the value counts sales since Tuesday's opening.

A value that only goes up can't drop on its own, so a drop can only mean the count started over. That gives you a rule: after a drop, the sample itself is the increase since the reset, because the new count started from zero. Everywhere else, the increase is the difference between two samples, as usual.

CalculationResultVerdict
Last minus first: 6 - 52-46The shop did not unsell 46 lattes
Sum of the per-sample changes: 3 + 3 + 2 + 412Reset-aware: at least 12 lattes were sold

At least, because the lattes sold after the 17:45 observation and before closing were never written down, and Tuesday's count doesn't contain them. More frequent observations shrink that hole. Nothing closes it.

You knew why the number dropped because you know the shop. A monitoring system doesn't know what a shop is. The samples carry a time, not the rule that opening zeroes the count, so the drop is the only signal it gets. And it's a fragile one: had Tuesday's first observation come late enough for the count to pass 58, there would be no drop, and the reset would go unnoticed.

Note

💡 You could carry Monday's total into Tuesday if you wanted to. A program can't carry anything across a restart: the count lived in its memory, and that memory is gone.

So an application "opens" every time it starts, and that happens on every:

  • deploy (a new version of the application replaces the running one)
  • crash (the program dies and gets restarted)
  • reboot of the machine it runs on

Every counter you will query has reset more than once, and never at 08:00.

Counters answer how much has happened. The next questions concern what is true now.

Gauges

A gauge represents a current value: the temperature in the room, the liters of milk in the fridge, the number of customers waiting at the counter.

Each sample replaces the previous one. The value can go up, go down, or stay where it is.

You suspect the weather has a say in how much coffee gets sold, so you start writing down the temperature as well. Here is the same hour as in the counters unit, this time from the thermometer by the window:

Timeindoor_temperature_celsiusWhat happened since the preceding sample
10:4520First sample
11:0024The sun reached the front window
11:1529The espresso machine ran non-stop through the rush
11:3022Somebody propped the door open
11:4523The door is closed again

The drop from 29 to 22 looks a lot like the counter reset from the previous unit. It isn't one.

For a counter, a drop meant the count started over and the raw values stopped being comparable. For a gauge, a drop is the measurement doing its job: the room got cooler.

CalculationResultVerdict
Latest sample23The current temperature
Last minus first: 23 - 203The room is warmer than at 10:45, rarely the question
Highest sample29The warmest the samples caught
Average of the samples: (20 + 24 + 29 + 22 + 23) / 523.6Typical temperature over the hour
Sum of the samples118Meaningless

Two things follow from how the 29 and the 23.6 were found:

  • A gauge keeps no history. The thermometer knows the current temperature and nothing else. The notebook holds only what you wrote down when you looked up.
  • The gap is lost here too, and it's worse. If the room hit 31 at 11:20 and cooled off before 11:30, no sample caught it. With a counter, sales made between two samples still show up in the next difference; only a reset loses them. With a gauge, a spike that rises and falls between two samples leaves no trace at all.

More frequent samples catch more of the movement. Once again, nothing eliminates the gap.

Drops are real, so minima, maxima, and averages are all on the table. Whether an operation answers anything still depends on the unit and the question:

OperationMeaningful?Why
Average of equally spaced temperature samplesYesIt summarizes the hour
Add 8 customers at the central shop and 5 at the riverside shopYes13 customers across both shops right now
Add 5 liters of milk in the fridge and 10 in the back roomYesSame unit, one inventory
Add 23 degrees inside and 15 degrees outsideNo38 is valid arithmetic and not a temperature anyone measured
Note

💡 In software, gauges are everywhere something has a current state: memory in use, open connections, items waiting in a queue, requests in flight.

Some are set from a fresh measurement, like the thermometer. Others move up and down as work starts and finishes, like a headcount at the door. Either way, the latest value is the state, and a drop is news rather than a reset.

Counters answer how much has happened. Gauges answer what is true now. The next unit turns the two behaviors into a decision, then lets you watch both in the cafe.

Choosing a Type

The right type comes from the question you will ask later, not from the measurement in front of you now.

Use this sequence:

  1. Are you counting events that accumulate? Use a counter. Ask how much the count changed over a period (the difference between the samples at its two ends), or how fast it changed (that difference divided by the length of the period). The raw lifetime total alone rarely answers anything.
  2. Are you measuring current state? Use a gauge. Decide whether the question needs the latest value, the highest or lowest sample, the average, or a sum, and whether the unit allows it.
  3. Neither? Some measurements fit neither behavior. How long each order took is the shop's example: no running total or latest value tells you how long orders usually take or how slow the slow ones are. What you want to know is how the durations spread out, and that needs a type of its own.
Note

💡 A different metric type exists for exactly this. We will explain it later in the course, where more practical examples make it easier to see how it works.

The shop's measurements now have clear semantics:

Future questionMeasurementTypeValid thinking
How many lattes were sold in the last hour?coffees_sold_total with drink="latte"CounterReset-aware increase over the hour
How many customers are inside now?customers_in_shopGaugeLatest value; sum across shops when useful
How much milk remains?milk_litersGaugeLatest value; sum compatible inventories
How warm was the room today?indoor_temperature_celsiusGaugeMin, max, or average over suitable samples
How long do orders usually take, and how slow are the slow ones?coffee_order_duration_secondsNeitherWill be covered later in the course

Watch Both Types Move

Start the playground for this lesson (if you haven't already), then open the Cafe tab.

Three things changed since the last lesson:

  • Observations are automatic now, every seven minutes of cafe time. One real second is one cafe minute, so that's every seven seconds for you.
  • A second thermometer hangs outside the door, so every entry records the outside temperature next to the inside one.
  • The notebook gained a Metrics view: the same entries sorted by metric, each one named as a counter or a gauge.

Sell a few coffees. Wait for an entry to appear. Sell a few more, and wait again. Keep going until the notebook holds at least five entries.

Now open Metrics and read the cards against the two units you just finished:

  • Coffees sold is a counter. It climbed each time you sold something and never came back down. What you sold between two entries shows up as the difference between them, and nothing else about those sales survives.
  • Inside temperature and Outside temperature are gauges. Each card shows the latest sample and a chart of what was written down. The chart goes both ways, and no drop on it is a reset.

Watch the thermometers on the left for a while. They move between entries, and a move that reverses before the next entry never reaches the chart. That's the gap from the gauges unit, live.

From Shop to Service

An order ticket and an HTTP access-log row both describe one completed event. The steps are the same as in the shop:

Coffee shopWeb service
The questionHow many lattes sold in the last hour? How many customers are being served right now?How many requests completed in the last hour? How many are in flight right now?
The behaviorSales accumulate, customers come and goCompleted requests accumulate, in-flight requests come and go

Here are the five requests from the previous lesson again, this time with how long each one took:

TimeRouteStatusDuration in seconds
09:00:01/checkout2000.18
09:00:03/checkout2000.24
09:00:04/checkout5000.78
09:00:08/menu2000.05
09:00:11/checkout5001.60

Count the completed requests by route and status, and you get the three labeled samples from the previous lesson. Now you can name their type: they are counters, one per route and status combination.

TimeMetric nameLabelsValue
09:01http_requests_totalroute="/checkout", status="200"2
09:01http_requests_totalroute="/checkout", status="500"2
09:01http_requests_totalroute="/menu", status="200"1

Each completed request increments exactly one of them.

An in-flight request gauge answers a different question: how many requests is the application processing right now? It increases when work starts and decreases when work finishes. You can't count that from the access log: a row appears only after the request has completed, so the log never shows what's still running. The application itself has to keep the number, adding one when a request starts and subtracting one when it finishes. That's the job of instrumentation, which the course covers in a later module.

Note

💡 Duration fits neither counter nor gauge.

The metric type for duration will be covered later in the course.

The mapping is direct:

Coffee shopWeb serviceType
Coffee saleCompleted HTTP requestCounter
Drink typeRoute or status breakdownLabel
Customers currently being servedRequests currently in flightGauge
Order durationRequest durationNeither, covered later in the course

You pick the type the same way for the shop and the service: from the question you'll ask later.

Summary

In this lesson, you learned how metric types describe how measurements were produced and which operations preserve their meaning.

Key takeaways:

  • Counter: coffees_sold_total with drink="latte" preserves accumulated latte sales so you can ask about reset-aware change over time
  • Gauge: customers_in_shop, milk_liters, and temperature preserve current state that may rise or fall
  • Neither: request duration fits neither type (will be covered later in the course)
  • Valid math: an operation is useful only when the metric type and the unit support the question: differences for counters, the latest, lowest, highest, or average sample for gauges, and sums only for values that share a unit
  • The same steps work for a web service: a coffee sale becomes a completed request (counter), customers being served become requests in flight (gauge), and order duration becomes request duration (neither).

That concludes the basics of the metrics data model. The following modules will focus on the practical aspects of exposing and collecting metrics.

Previous lesson
Labels and Series