What Is Observability?
Production Is Invisible
It's Friday, 16:45. A deploy went out an hour ago. Checkout is now returning 500 errors to a number of users, and you have no idea about what's going on.
Nothing crashed. No process died. The service is still running, but it is quietly losing money, and you find out because a customer tweets about it.
A doctor faces a similar problem with a body. They can't directly see every process happening inside, so they measure it (temperature, heart rate, a blood panel) and reason backward from those results to what's going wrong. A production service is similar: a process can be running while user requests fail. You can observe some behavior from outside, but understanding its internal state requires evidence collected from the application.
That's the uncomfortable truth underneath the Friday scenario: a running system's internal state is largely invisible by default. Useful evidence has to be collected deliberately.
Data collected about a system's behavior is called telemetry. Some comes from application instrumentation (code or configuration that emits this data). The surrounding platform also contributes telemetry: runtimes execute application code, hosts run processes, proxies route traffic, and orchestrators manage deployed workloads. External checks collect telemetry from outside the system.
Without useful evidence, diagnosis is guesswork. You're the doctor with no thermometer, guessing.
This is what observability is for: inferring what's happening inside a system from the signals it puts out.
What "invisible" costs you
When no telemetry is collected, every part of incident response becomes slower and less reliable:
| You want to... | Without useful telemetry | With useful telemetry |
|---|---|---|
| Detect that something is wrong | Depend on customer reports or manual checks | A configured alert can flag a condition represented in the data |
| Diagnose what's happening | SSH in and inspect components one by one | Inspect the relevant evidence |
| Decide whether to act | Guess from sparse evidence | Compare the impact with normal behavior |
| Verify recovery | "Seems fine now?" | Check that the affected signals return to normal |
"But we have logs...somewhere."
If that's what you're thinking right now, it doesn't get you out of this. Logs you can't find, query, or correlate contribute little during an incident. They're evidence locked in a room you don't have the key to.
The point isn't that data exists. It's that the system was built to surface useful data, in a shape you can reason from, before you needed it.
Where the word "observability" comes from
The word predates today's observability products. In 1960, engineer Rudolf E. Kálmán formalized observability in control theory. For a specified dynamical system, complete observability means that its state can be uniquely determined from known inputs and observed outputs over a finite interval.
Software observability adapts that idea rather than applying the mathematical test directly. The practical question for a running service is similar: does the evidence the system exposes let you understand its internal behavior?
And the catch that makes it hard in software: useful visibility doesn't happen by accident. Observability is something you build through instrumentation and external collection, not something you get merely because a process is running.
Telemetry Signals
In the opening incident, a customer report gave you only the symptom: some users were receiving 500 errors. Diagnosing the failure requires asking a range of questions.
The answers come from evidence collected about the service.
For example:
| Question | Answer |
|---|---|
| How bad is it? | 5% of requests are failing, request latency doubled |
| What exactly happened? | payment-service returned connection refused at 16:03 |
| Where did the request spend its time? | 1.8 of 2.1 seconds waiting on the payments call |
| Which code path is the slowest? | A profiler found JSON serialization active in 40% of its observations |
No single form of data is best at answering all four questions. A number that says "5% failing" does not contain the exact error for one request. An error record may include a duration, but it may not show how work was divided across several services. That's why telemetry comes in different shapes.
Telemetry is data collected about a system's behavior, whether the source is the application itself, the platform around it, or an external observer. The signals differ not by which vendor sells them, but by the shape of evidence they store.
Four common signal types
| Signal | Data shape | Typical strength | Main trade-off |
|---|---|---|---|
| Metrics | Numerical measurements tracked over time, often combining many events into counts and distributions | Measuring aggregate health, size, and trends | Compression discards per-event detail; too many categories can make metrics expensive |
| Logs | Timestamped records of events with context | Explaining what happened during an individual event | Event volume and retained context can make storage and queries expensive |
| Traces | Correlated records of steps, called spans, that represent one operation | Showing how work and time were divided across components | Only instrumented operations and recorded spans are visible |
| Profiles | Samples or instrumented runtime events that associate resource use with active code paths | Finding functions and chains of calls that consume resources | Both methods capture selected observations rather than a complete execution history |
Choose a signal by the question, not by its position in the table:
- To size a problem across many requests, start with metrics.
- To inspect one event and its recorded context, reach for logs.
- To follow one operation across components, inspect a trace.
- To find which code paths consume a resource (eg. leaking memory), inspect a profile.
These are strengths, not hard boundaries. A structured log can include a duration, a trace can record an error, and a metric can be linked to richer evidence. The signals are often most useful when you can correlate them.
The trap to avoid
Each signal preserves some detail and discards other detail. Collecting more data does not automatically preserve the context your question needs, and retaining everything is neither practical nor useful.
The rest of this course is about metrics specifically: a signal well suited to repeated questions such as "how many?", "how bad?", and "is it changing?" Before explaining why metrics come first, the next unit distinguishes monitoring from observability.
Signals
You'll often see observability described as three pillars: metrics, logs, and traces. It's a popular framing, but treat it as a beginner map, not a law.
This course mentions profiles because they answer a distinct class of resource-use questions. That makes metrics, logs, traces, and profiles four common signal types, not a universal list or four mandatory components. Events and exceptions can be represented within logs or traces, while continuous profiling is a way to collect profiles.
We'll often say signals instead of pillars in this course. It doesn't imply you've failed until you have all of them: a setup using only one signal type can still be useful.
Monitoring vs Observability
Suppose the checkout service had recorded its error rate. A graph could have shown the spike, and an alert could have notified you because someone had decided in advance to watch that condition. That's monitoring: the practice of collecting and analyzing system data to track health, spot trends, investigate behavior, and act on conditions you care about.
The graph tells you 5% of checkouts are failing, and you ask a question nobody wired into the alert: is it all users, or just some? If the data preserved useful categories, you could break down the failures and discover that they affect only certain users paying in EUR. This is where observability matters: the system exposed enough evidence to investigate beyond the original condition.
Monitoring is an operational practice. Observability is a property of the system and its telemetry. They aren't rivals, and they are not separated by "predefined query" versus "new query." Monitoring uses the available evidence; a more observable system makes monitoring and investigation more effective.
Practice vs property
| Monitoring | Observability | |
|---|---|---|
| What it is | An operational practice | A property of a system and its telemetry |
| The question it asks | What data are we collecting and analyzing? | What can the available evidence reveal? |
| Examples | Creating dashboards, alerting, analyzing trends, and running investigations | Evidence that can explain known conditions and unexpected behavior |
| Limited by | Collection coverage and analysis | The detail and context the system exposed |
The reason the distinction matters isn't philosophical. It's that different data shapes preserve different detail, and you can't recover detail that a dataset discarded.
Why Metrics First
This course spends almost all its time on one of the four common signal types introduced here. Why start with metrics instead of logs, traces, or profiles?
Because well-designed metrics are often a compact and efficient way to answer aggregate questions.
Why not logs?
Logs probably came first in your applications. Most application frameworks include logging, so applications often emit basic logs before anyone adds metrics. Those logs are useful evidence, but useful logging still takes deliberate work: choosing structure and context, setting levels, collecting records, deciding retention, and making the data queryable.
Rather than perfecting those logs first, pair them with a small set of well-designed metrics. Metrics summarize behavior across many events and can drive dashboards and alerting rules. Logs preserve the detail of individual events. A metric can narrow the relevant time window and part of the system, while the logs help explain what happened there. The questions neither can answer then guide how you improve both.
This combination gives you useful coverage sooner, and often at lower cost than trying to derive every repeated aggregate from logs. Logs retain records for individual events, so collecting, storing, and repeatedly querying them at high volume can be expensive. Well-designed metrics aggregate those events into compact time series.
In our incident, request metrics could have shown one useful summary: 5% of checkouts are failing, up from 0.1%. A configured alerting rule could have triggered an alert that paged you, the number would have sized the impact, and the same measurement could have helped verify recovery. Finding the exact cause would still require follow-up evidence.
What metrics are unreasonably good at
Metrics represent numerical measurements over time. Many aggregate, or combine, repeated events into counts, ratios, or distributions; others record a current state, such as queue depth. Their compact numerical shape is both the strength and the limit.
The strength:
- Often compact. A collection interval controls how often the monitoring system collects metric values. Longer intervals generally produce fewer stored data; they do not make application instrumentation record fewer events. With categories limited to a known set of values, metrics can summarize large volumes of behavior in relatively little space.
- Optimized for repeated numerical questions. Metric systems are designed to calculate rates, ratios, trends, and aggregations repeatedly for dashboards and alerting rules.
- Directly useful. Metrics commonly power dashboards, alerts, and capacity planning, which estimates how much traffic or resource demand a system must handle. For a success metric, a reliability target states the success ratio the service should maintain, giving the observed ratio a goal to compare against.
- They force clarity. To create a metric you have to name what matters. "What number would tell me checkout is healthy?" is a more useful question than most teams ask before an outage.
That combination makes metrics a common starting point and the right place to start this course. It is not a universal cost law: a poorly designed metric with too many categories can be expensive.
Where metrics stop
The same compact representation that makes well-designed metrics useful also throws away detail. Request rate and error rate can reveal a broad problem soon after data is collected, but a typical aggregate metric does not retain the exact request payload, stack trace, or user ID. That detail may still exist in a correlated log or trace.
| Question | Metric alone | Better signal |
|---|---|---|
| How many checkouts are failing? | Yes, from aggregate counts | |
| Is it getting worse? | Yes, from the trend | |
| What was the exact error for one request? | Usually not | Logs |
| Where did one slow request spend its time? | Usually not | Traces |
| Which function is burning the CPU? | Usually not | Profiles |
So "metrics first" is a claim about this course's sequence, not sufficiency. Metrics are well suited to showing that something is wrong and how bad. They direct the investigation; logs, traces, and profiles can supply finer context.
That's the frame for the rest of the course. You're learning a signal with broad operational reach and relatively compact data when it is designed well, while staying clear-eyed about the questions it will hand off to the others.
- Next lesson
- How This Course Works