Deploy an observability stack

In the two previous lessons you saw the two gaps:

• Events are deleted after an hour.
• kubectl top has no history, knows nothing about your application and warns you of nothing.

Both gaps are closed by the same thing, and it is not a product: it is an architecture. Someone collects the signal, someone stores it, and someone queries it.

In this cluster, Prometheus (metrics), Loki (logs) and Grafana (the window onto both) are already installed. Your job is not to install them: it is to connect your application and understand why it works.

kubectl -n observabilidad get pods

kubectl -n tienda get all

Step 1: Prometheus is not a service, it is an operator

Before touching anything, look at what the chart has installed:

kubectl api-resources --api-group=monitoring.coreos.com

There they are: Prometheus, ServiceMonitor, PodMonitor, PrometheusRule, Alertmanager...

You will recognize the pattern, because you built it yourself in the Extensibility module. The Prometheus Operator is exactly the same loop you wrote: it watches ServiceMonitor objects, and when a new one appears, it regenerates the Prometheus configuration and reloads it. Nobody is editing prometheus.yml by hand.

This has a consequence that changes the way you work: adding an application to monitoring stops being a platform team's task. The development team applies a ServiceMonitor in its namespace, next to its Deployment, in its repository. Prometheus finds out on its own.

Step 2: Connect your application

You need two objects, and here is where 90% of the mistakes of people starting with Prometheus happen.

Create monitorizacion.yaml:

cat << 'EOF' > monitorizacion.yaml
apiVersion: v1
kind: Service
metadata:
  name: web-metrics
  namespace: tienda
  labels:
    app: web
spec:
  selector:
    app: web
  ports:
  - name: metrics        # <- THE NAME MATTERS
    port: 80
    targetPort: 80
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: web
  namespace: tienda
  labels:
    app: web
spec:
  selector:
    matchLabels:
      app: web           # <- selects SERVICES, not Pods
  endpoints:
  - port: metrics        # <- by NAME, not by number
    path: /metrics
    interval: 15s
EOF

The YAML, explained in questions and answers

Why two objects and not one?

Because a ServiceMonitor does not select Pods: it selects Services. Prometheus reaches your Pods through the EndpointSlices of a Service. The ServiceMonitor is a configuration layer on top of a Service that already exists.

It is the number one cause of "I created the ServiceMonitor and Prometheus sees nothing": there was no Service, or the Service had no endpoints.

Why is the port referenced by name (port: metrics) and not by number?

Because the ServiceMonitor demands it. spec.endpoints[].port is the name of the Service's port, not the number. If you write port: 80 there, Prometheus looks for a port called "80", does not find it, and gives no error at all: it simply does not generate the target.

That silence is what turns this mistake into a lost afternoon. If your ServiceMonitor does not show up in /targets, check this field first.

(There is spec.endpoints[].targetPort for rare cases, but the correct and robust way is to name the port in the Service and reference it by name.)

What is selector.matchLabels? Yet another one?

Yes, and you need a clear map, because in Kubernetes there are chained selectors everywhere:

ServiceMonitor --(selector)--> Service --(selector)--> Pods

The ServiceMonitor selects the Service by its labels. The Service selects the Pods by theirs. If either of the two links fails, there are no metrics and there is no error.

And how does Prometheus know it should read this ServiceMonitor rather than ignore it?

Good question, and it is the second cause of failure. The Prometheus object has a serviceMonitorSelector: by default, it only reads the ServiceMonitors carrying certain labels. In this playground we have opened it up on purpose (serviceMonitorSelectorNilUsesHelmValues=false), so that Prometheus picks up any ServiceMonitor in the cluster.

In production you will not want that: you will want to make sure a team cannot bring down the central Prometheus by applying a ServiceMonitor that scrapes ten thousand endpoints every second.

What if my application does not expose /metrics?

Then there is nothing to collect, and no tool in the world fixes that. Prometheus does not guess: your application has to be instrumented and expose an endpoint in Prometheus format. It is a library (prometheus-client in Python, client_golang in Go...) and a few lines of code.

What you do get for free, without touching the application, are the infrastructure metrics: kube-state-metrics gives you the state of every object (replicas, restarts, phases), and node-exporter the state of the nodes. Both come with the chart. What you do not get for free is "how many orders per second my tienda processes".

Apply it:

kubectl apply -f monitorizacion.yaml

kubectl -n tienda get endpointslice -l kubernetes.io/service-name=web-metrics

Step 3: Watch Prometheus discover it on its own

You have not touched Prometheus. You have not restarted anything. And yet:

Open the Prometheus tab and go to Status → Targets. In under a minute your target appears, in green, UP.

kubectl -n observabilidad port-forward svc/kps-kube-prometheus-stack-prometheus 9090:9090

The operator did that: it saw your ServiceMonitor, regenerated the configuration, and told Prometheus to reload it.

Try a PromQL query in /graph:

up{namespace="tienda"}

And now, the ones you will really use. These come from kube-state-metrics and work without instrumenting anything:

# Container restarts in the last hour: the CrashLoop detector
rate(kube_pod_container_status_restarts_total[1h]) > 0

# Actual CPU per Pod
sum(rate(container_cpu_usage_seconds_total{namespace="tienda"}[5m])) by (pod)

# What kubectl top could not tell you: ACTUAL usage versus what is RESERVED
sum(rate(container_cpu_usage_seconds_total{namespace="tienda"}[5m])) by (pod)
  / sum(kube_pod_container_resource_requests{namespace="tienda", resource="cpu"}) by (pod)

That last query is the rightsizing of the previous lesson, but with history and over the whole cluster at once. This is where you see what the stack brings.

Step 4: The logs, with Loki

Loki is deployed, and Promtail (its agent) runs as a DaemonSet: one Pod per node, reading the log files of every container on the node.

kubectl -n observabilidad get daemonset

That DaemonSet is exactly the pattern you studied in the Workloads module, and it is the log collection pattern: the agent goes to the node, not to the application. Your application keeps writing to stdout and does not know Loki exists. That decoupling is what makes the system scale.

Open the Grafana tab (admin / admin), go to Explore, choose the Loki data source and query:

{namespace="tienda"}
{namespace="tienda"} |= "GET"
{namespace="observabilidad"} |= "error"

The idea to take away: Loki does not index the text of the logs, only the labels (namespace, pod, container). Those labels are the same ones Kubernetes uses. That is why you can jump from a metric to the logs of the same Pod without switching mental language: the correlation is not magic, it is that both tools use the Kubernetes object model as their coordinate system.

Step 5: Your first alert

Here is the third gap kubectl top left: nobody warns you.

An alert in Prometheus is, once again, a CRD. Create alertas.yaml:

cat << 'EOF' > alertas.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: web-alertas
  namespace: tienda
  labels:
    app: web
spec:
  groups:
  - name: web.reglas
    rules:
    - alert: WebCaida
      expr: up{namespace="tienda"} == 0
      for: 1m
      labels:
        severity: critical
      annotations:
        summary: "La web de la tienda no responde"
        description: "El endpoint {{ $labels.instance }} lleva 1 minuto sin responder."

    - alert: WebDesaparecida
      expr: absent(up{namespace="tienda"})
      for: 1m
      labels:
        severity: critical
      annotations:
        summary: "La web de la tienda ha desaparecido del radar"
        description: "Prometheus ya no tiene ningun target en el namespace tienda."

    - alert: PodReiniciandoEnBucle
      expr: rate(kube_pod_container_status_restarts_total{namespace="tienda"}[10m]) > 0
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "El Pod {{ $labels.pod }} se está reiniciando en bucle"
EOF

The alert names and texts are in Spanish, like the rest of the tienda: WebCaida is "web down" ("the tienda's web is not responding"), WebDesaparecida is "web gone" ("the tienda's web has disappeared from the radar", "Prometheus no longer has any target in the tienda namespace"), and PodReiniciandoEnBucle is "Pod restarting in a loop".

What does for: 1m do? It is the difference between an alerting system and a noise generator. The condition has to hold continuously for that long before the alert fires. Without it, a two-second spike wakes you up at four in the morning. With it, a normal restart during a rollout bothers nobody.

And why two alerts for the same thing? They are not the same thing, and this is probably the trap that has caused the most silent alerts in the history of Prometheus. Hold on to the answer for a moment: you are going to check it yourself.

kubectl apply -f alertas.yaml

kubectl -n tienda get prometheusrule

Check it in Prometheus, Alerts tab. Your three rules are there, in green. And now take the web down:

kubectl -n tienda scale deployment web --replicas=0

Wait a minute and look at the Alerts tab again.

What just happened: == 0 is not the same as "no data"

WebDesaparecida has fired. WebCaida is still green, even though the web is as down as it can be. If that were your only alert, nobody would have warned you.

The reason lies in how Prometheus builds its list of targets. The targets come from the Service's EndpointSlice, and those endpoints are the Pods' IPs. When you scale to 0 no Pod is left, so no endpoint is left, so no target is left.

And without a target there is no up metric. The series is not 0: it stops existing.

web with 2 Pods   →  up{...} = 1, up{...} = 1     →  `up == 0` no match: evaluates to empty
web not answering →  up{...} = 0                  →  `up == 0` matches: ALERT
web scaled to 0   →  (there is no `up` series)    →  `up == 0` evaluates to empty: SILENCE

up == 0 is a filter over the series that exist. If none exists, the filter returns empty, and a rule whose expression returns empty is not firing: it is off. That is why you see it evaluated over and over without changing state.

absent() is the function that exists precisely for this: it returns 1 when its argument returns nothing, and nothing when it does return something. It is the only way to alert on the absence of a signal.

The pair up == 0 + absent(up) covers the two failures, which are different and are fixed differently: "it is there and not answering" (the process has died, the port is closed) and "it is gone" (someone deleted the Deployment, the Service was left without a selector, the ServiceMonitor stopped matching). The second is the more dangerous of the two, because a monitoring system that falls over by itself does not complain.

Bring it back to life and watch both go green again:

kubectl -n tienda scale deployment web --replicas=2

And the third pillar: traces

Traces are missing, and you are not going to set them up here because they need something a lab cannot give you: a real application, with several services calling each other.

The idea, in one sentence: a trace follows one specific request through every service it passes through, and tells you where the time went. Metrics tell you the p99 has gone up to 3 seconds; the trace tells you that 2.8 of those seconds were eaten by a database query, in the third service of the chain.

Today the standard is OpenTelemetry, and its promise is that instrumentation stops being proprietary: you instrument once, and then decide where to send the data (Jaeger, Tempo, or whichever SaaS). You change the backend with a configuration file; the application code is untouched.

And here you do have to be honest about the cost: you can get metrics and logs without touching your application (infrastructure metrics with kube-state-metrics, logs with a DaemonSet). Traces, no. They require instrumenting the code, propagating headers between services, and having every service in the chain cooperate. A single uninstrumented service breaks the trace. That is why it is the pillar most people postpone, and the one they miss the most on the day of the strange incident.

Summary

• An observability stack is three functions: collect, store, query. Prometheus, Loki and Grafana are one possible implementation.
• The Prometheus Operator is the operator pattern you built yourself: it watches ServiceMonitor and PrometheusRule, and regenerates the Prometheus configuration. Nobody edits files by hand.
• A ServiceMonitor selects Services, not Pods, and references the port by name. The two most common mistakes, and neither of them gives an error: only silence.
• kube-state-metrics and node-exporter give you infrastructure observability for free. Business metrics require instrumenting your application.
• Logs are collected with one DaemonSet per node. The application writes to stdout and knows nothing about it. (Promtail is end of life: today you use Alloy or the OpenTelemetry Collector.)
• An alert is a CRD, and its most important field is for: without it you have a noise generator, not a warning system.
• Traces are the only pillar that requires touching the code, and the standard for doing it is OpenTelemetry.

You have just set up a stack that works. Before calling it done, look at what it costs:

kubectl top pods -n observabilidad --sort-by=memory

That Prometheus is storing two hours of metrics from a cluster with three Pods, and you can already see it in the memory. With retention=15d and a real cluster, that number grows fast, and it grows in RAM: Prometheus keeps the recent blocks and an inverted index of every active series in memory.

That is not a defect of Prometheus. It is the result of a deliberate design decision: Prometheus is meant to be ephemeral, local and not distributed. Its own documentation says so bluntly: it is not durable long-term storage.

And that is where another piece comes in.

Step 1: Install VictoriaMetrics

VictoriaMetrics is a Prometheus-compatible time series database. It is not a replacement for Prometheus as such: it is a replacement for its storage, with a different implementation designed to last and to take up less space.

helm upgrade --install vm vm/victoria-metrics-single \
  --namespace observabilidad \
  --set server.fullnameOverride=vmsingle \
  --set server.retentionPeriod=1 \
  --set server.service.type=NodePort \
  --set server.service.nodePort=30428 \
  --wait

kubectl -n observabilidad get pods -l app.kubernetes.io/name=victoria-metrics-single

kubectl -n observabilidad get svc vmsingle

A single Pod. A single binary, with no external dependencies: that is single-node mode, and it holds up a good deal more than people assume (millions of active series on a single machine). The cluster mode (vminsert / vmstorage / vmselect, three components that scale separately) exists for when this gets too small, and that separation is precisely what Prometheus does not give you.

Step 2: Connect Prometheus with remote_write

Here is the architecture that is actually used in production, and it is worth understanding before typing it:

  ServiceMonitor  ──▶  Prometheus  ──remote_write──▶  VictoriaMetrics
                       (collects,                        (stores,
                        short retention)                long retention)

Prometheus keeps doing what it does best: discovering targets and scraping them. But on top of that, every few seconds, it forwards everything it collects somewhere else using the remote_write protocol. Prometheus keeps a few hours for the alerts; VictoriaMetrics keeps the months.

And this is not configured by editing any file. The Prometheus object is a CRD, and remoteWrite is a field of its spec:

kubectl -n observabilidad patch prometheus kps-kube-prometheus-stack-prometheus \
  --type=merge \
  -p '{"spec":{"remoteWrite":[{"url":"http://vmsingle.observabilidad.svc:8428/api/v1/write"}]}}'

Look at the operator's logs if you want to see it at work. It detected the change in the CR, regenerated the prometheus.yml and told Prometheus to reload it. It is exactly the loop you wrote in the Extensibility module, now governing a critical piece of your platform.

Step 3: Ask VictoriaMetrics

Now the part that makes all of this worthwhile.

Open the VictoriaMetrics tab and go to /vmui. It is VictoriaMetrics' own interface. Type the same query you typed in Prometheus:

up{namespace="tienda"}

sum(rate(container_cpu_usage_seconds_total{namespace="tienda"}[5m])) by (pod)

They work. Without changing a comma.

Why does it work?

Because VictoriaMetrics speaks the Prometheus protocol, at both ends:

• Write: it accepts remote_write, which is the standard mechanism by which Prometheus exports its samples.
• Read: it exposes /api/v1/query and /api/v1/query_range, exactly the same endpoints as Prometheus. Any client that knows how to talk to Prometheus knows how to talk to VictoriaMetrics without knowing it exists.

This is a lesson that goes well beyond this tool: in observability, the Prometheus interface has become the de facto standard. Thanos, Mimir, Cortex, VictoriaMetrics... they are all different pieces inside and they all behave like a Prometheus outside. It is what lets you swap the engine without changing a single alert, a single dashboard, or a single line of your application.

Step 4: Add it to Grafana

Check it where it matters. Grafana discovers its data sources by reading labeled ConfigMaps (a sidecar the chart deploys does it; look at it with kubectl -n observabilidad get pods -l app.kubernetes.io/name=grafana -o jsonpath='{.items[0].spec.containers[*].name}').

Create vm-datasource.yaml:

cat << 'EOF' > vm-datasource.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: vm-datasource
  namespace: observabilidad
  labels:
    grafana_datasource: "1"      # <- without this, the sidecar ignores it
data:
  vm.yaml: |
    apiVersion: 1
    datasources:
    - name: VictoriaMetrics
      type: prometheus           # <- read this twice
      url: http://vmsingle.observabilidad.svc:8428
      access: proxy
      isDefault: false
EOF

kubectl apply -f vm-datasource.yaml

Wait a few seconds (the sidecar reloads on its own), open the Grafana tab, go to Explore, and choose the VictoriaMetrics source. Run any of the earlier queries.

That type: prometheus is the entire moral of this unit in one line. You have not integrated VictoriaMetrics with Grafana. You have told Grafana there is a Prometheus at that URL, and VictoriaMetrics has behaved like one.

Step 5: MetricsQL, and where the fine print is

VictoriaMetrics does not run PromQL: it runs MetricsQL, which is a superset. All your PromQL works, and on top of that there are shortcuts PromQL does not have:

# In PromQL, this is an error: rate() requires a window.
rate(container_cpu_usage_seconds_total{namespace="tienda"})

# In MetricsQL, the window is inferred from the graph's interval.

# And functions that in PromQL take some juggling:
rollup_rate(container_cpu_usage_seconds_total[5m])   # min, max and avg in one go

Here it pays to be honest, and it is the kind of thing a book should tell you:

The rest of the family

What you have set up is the central piece, but the project has more:

• vmagent: a scraper that can replace Prometheus entirely for collection. It understands the same ServiceMonitors, uses considerably less memory, and can buffer to disk if the destination does not answer. In the setup you have built, Prometheus is still the one collecting; with vmagent, Prometheus disappears from the diagram.
• vmalert: runs alerting and recording rules against VictoriaMetrics, with the same syntax as PrometheusRule. It is what you need if you remove Prometheus, because the alerts went with it.
• VictoriaMetrics Operator: its own CRDs (VMServiceScrape, VMRule, VMAlert...) and a converter that reads the ServiceMonitors and PrometheusRules you already have. The same thing again: the ecosystem has agreed that the ServiceMonitor is the contract.
• VictoriaLogs: the equivalent for logs, in the same space as Loki.

In other words: you can replace the whole stack, or only the piece that hurts. That gradual path is the whole pitch.

So, Prometheus or VictoriaMetrics?

The question is badly framed, and that is the lesson.

Prometheus defines the standard. It is the CNCF graduated project, it is what everyone knows how to read, and its data model and its API are the common language of observability. You are not abandoning it: you are implementing it with another engine.

VictoriaMetrics is an implementation of that standard optimized to last longer and take up less. And it is not the only one: Thanos and Mimir solve the same problem with different architectures.

The useful question is not which is better, but when stock Prometheus stops being enough for you:

If you do not need the last three rows, do not set this up. Adding a distributed database to a cluster that does not need it is an expensive way of having more things that break.

And when you do need it, you already know the best part: your alerts, your dashboards and your applications will not notice.

Summary

• Prometheus is designed to be ephemeral and local. Its storage is not the place to keep a year of metrics, and it does not pretend to be.
• remote_write is the standard mechanism for sending the samples somewhere else: Prometheus collects, something else stores. In the operator it is a field of the CRD, not a file.
• VictoriaMetrics implements the Prometheus API at both ends (write and read). That is why Grafana declares it with type: prometheus and it works without touching a query.
• The Prometheus interface is the de facto standard of observability. VictoriaMetrics, Thanos and Mimir are different engines behind the same facade. You swap the engine without changing anything else.
• MetricsQL is a superset of PromQL: it gives you extra functions, and it ties you down if you use them. Stay on standard PromQL unless you have a reason.
• The full family (vmagent, vmalert, the operator with its ServiceMonitor converter) lets you replace the stack piece by piece, not all at once.
• And the most important answer: if you do not have a retention, scale or high-availability problem, do not change anything.