Getting Started with VictoriaMetrics on Kubernetes

VictoriaMetrics has a dedicated Kubernetes Operator that handles deploying and managing VictoriaMetrics components on Kubernetes, as well as configuring scrape targets for your applications (and even for Kubernetes itself).

In this tutorial, you will learn how to:

• Install the VictoriaMetrics Operator
• Deploy a single-node VictoriaMetrics instance
• Deploy vmagent
• Configure Kubernetes metrics scraping
• Query cluster metrics from the terminal

Installing the Operator

The VictoriaMetrics Operator publishes ready-to-use Kubernetes manifests on its GitHub releases page.

Set the version to install:

VM_OPERATOR_VERSION=v0.68.3

Apply the manifest to install the operator, its CRDs, and the required RBAC resources:

kubectl apply -f "https://github.com/VictoriaMetrics/operator/releases/download/${VM_OPERATOR_VERSION?}/install-no-webhook.yaml"

This creates the vm namespace and deploys the operator in it.

Wait for the operator to become ready:

kubectl -n vm wait --for=condition=Available deployment vm-operator

Deploying a Single-Node Instance

Now that the operator is running, take a look at the Custom Resource Definitions (CRDs) it installed:

kubectl api-resources | grep victoriametrics

Each CRD represents a different VictoriaMetrics component or configuration that the operator knows how to manage. You'll use three of them in this tutorial:

• VMSingle
• VMAgent
• VMNodeScrape

The first one you need is VMSingle. When you create a VMSingle resource, the operator provisions:

• a Deployment running a single-node VictoriaMetrics instance
• a PersistentVolumeClaim for data storage (if you configure storage)
• and a Service to expose it within the cluster.

Create a VMSingle resource:

kubectl apply -f - <<EOF
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMSingle
metadata:
  name: victoriametrics
spec:
  retentionPeriod: "7d"
  storage:
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: 2Gi
EOF

Wait for the instance to become ready:

kubectl wait --for=condition=Available deployment vmsingle-victoriametrics

The operator also created a Service for the instance:

kubectl get svc vmsingle-victoriametrics

This is the endpoint that other components will use to push metrics to VictoriaMetrics.

To make the VictoriaMetrics API accessible outside the cluster, create an Ingress resource:

kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: vmsingle-victoriametrics
spec:
  rules:
    - http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: vmsingle-victoriametrics
                port:
                  number: 8428
EOF

Verify you can access VictoriaMetrics from the terminal:

curl http://cplane-01/health

Deploying vmagent

VictoriaMetrics stores metrics, but it doesn't (usually) collect them on its own. That's the job of vmagent, a lightweight agent that scrapes metrics from targets and pushes them to VictoriaMetrics via remote write.

The VMAgent CRD tells the operator to create a Deployment running vmagent, preconfigured with the scrape targets and remote write destination you specify.

Create a VMAgent resource:

kubectl apply -f - <<EOF
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAgent
metadata:
  name: vmagent
spec:
  selectAllByDefault: true
  replicaCount: 1
  remoteWrite:
    - url: "http://vmsingle-victoriametrics.default.svc.cluster.local:8428/api/v1/write"
EOF

Wait for vmagent to become ready:

kubectl wait --for=condition=Available deployment vmagent-vmagent

Configuring Kubernetes Metrics Scraping

vmagent is running, but it has no idea what to scrape yet.

If you've worked with Prometheus before, you might expect to write raw scrape_configs YAML.

The operator takes a different approach: you define what to scrape using CRDs, and the operator translates them into scrape configuration for vmagent automatically.

If you're familiar with the Prometheus Operator, these work like ServiceMonitors and PodMonitors, but are managed by the VictoriaMetrics Operator instead.

Since you want to monitor the Kubernetes cluster itself, VMNodeScrape is the right choice.

Create two VMNodeScrape resources to collect kubelet and cadvisor metrics from every node:

kubectl apply -f - <<EOF
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMNodeScrape
metadata:
  name: kubelet-metrics
spec:
  scheme: https
  tlsConfig:
    caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token

---
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMNodeScrape
metadata:
  name: kubelet-cadvisor
spec:
  scheme: https
  path: /metrics/cadvisor
  tlsConfig:
    caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
EOF

vmagent automatically picks up these CRDs (thanks to selectAllByDefault: true) and starts scraping the kubelet and cadvisor endpoints on every Node.

kubectl get secret vmagent-vmagent -o jsonpath={'.data.vmagent\.yaml\.gz'} | base64 --decode | gunzip | yq

Querying Metrics

Give vmagent a few seconds to scrape and push data, then query:

curl -s 'http://cplane-01/api/v1/query?query=up' | jq

You should see targets with "__name__":"up" and a value of 1 for each healthy scrape target.

Try some Kubernetes-specific queries:

Number of running Nodes:

curl -s 'http://cplane-01/api/v1/query?query=count(kubelet_node_name)' | jq

Container memory usage:

curl -s 'http://cplane-01/api/v1/query?query=sum(container_memory_working_set_bytes)by(pod)' | jq

Summary

Congratulations! You've deployed a full VictoriaMetrics monitoring stack on Kubernetes using the VictoriaMetrics Operator.

Here's what you covered:

• VictoriaMetrics Operator manages VictoriaMetrics components through CRDs
• VMSingle deploys a single-node VictoriaMetrics instance with persistent storage
• VMAgent deploys vmagent with automatic scrape target discovery
• VMNodeScrape replaces raw scrape configs with declarative Kubernetes resources
• Scrape discovery is automatic: vmagent picks up new CRDs without restarts

References

• Operator documentation
• VictoriaMetrics Helm charts