The operator

In the previous lesson you taught Kubernetes a noun (Promocion) and checked that it meant nothing. Now you are going to give it a verb, and you are going to write it in the same language and with the same library that Kubernetes itself uses.

A controller is a program that does three things, in a loop, forever:

1. Observe the desired state.
2. Compare it with the current state.
3. Act to bring the second closer to the first.

The kube-controller-manager you studied in the Kubernetes architecture chapter is nothing more than a pile of these loops running together: one for Deployments, another for ReplicaSets, another for nodes. Yours is going to be one more, and it is going to use exactly the same library: controller-runtime.

The code is already written, in /home/laborant/operador. The goal of this unit is not to type it: it is to read it, so open it in the IDE tab.

types.go: the CRD, in Go

The CRD you applied earlier is a contract in YAML. This file is the same contract in Go:

var GroupVersion = schema.GroupVersion{Group: "tienda.example.com", Version: "v1"}

type PromocionSpec struct {
    Banner   string `json:"banner"`
    Replicas *int32 `json:"replicas,omitempty"`
}

type PromocionStatus struct {
    ReadyReplicas int32  `json:"readyReplicas,omitempty"`
    Fase          string `json:"fase,omitempty"`
}

type Promocion struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`

    Spec   PromocionSpec   `json:"spec,omitempty"`
    Status PromocionStatus `json:"status,omitempty"`
}

Questions and answers

Why is Replicas a pointer (*int32) and Banner is not?

Because a pointer can be nil, and an empty string cannot be told apart from "it was never set". With *int32, the operator can tell the difference between "the user asked for 0 replicas" and "the user said nothing". With a plain int32, both cases arrive as 0 and are indistinguishable.

It is the convention across the whole Kubernetes API: if a field is optional and its zero value means something, it goes as a pointer. Look at a Deployment's spec.replicas: it is also *int32.

What are the embedded TypeMeta and ObjectMeta?

They are the apiVersion/kind and the metadata that every Kubernetes object carries. By embedding them, your type inherits Name, Namespace, Labels, Annotations, OwnerReferences... and becomes a fully fledged Kubernetes object, not just any data structure.

And those DeepCopy functions at the end of the file?

That is the one concession in this lesson. Kubernetes requires every object to know how to deep-copy itself, because clients cache objects in memory and a shallow copy would let a controller accidentally modify the shared cache. It is a memory-safety requirement, not a whim.

In a real project those functions are generated by controller-gen from a few magic comments (// +kubebuilder:object:root=true). Here we have written them by hand, and they are twenty lines, because the goal is for this operator to have not a single line of magic.

What does AddToScheme do?

It registers your type in the scheme: the table that tells the client "when you see an object with apiVersion: tienda.example.com/v1 and kind: Promocion, deserialize it into this Go struct, and to write it, send the request to /apis/tienda.example.com/v1/namespaces/{ns}/promociones".

Without that registration, the client would not even know which URL to call.

main.go: the reconciler

Here is the heart. The Reconcile function is called once for every Promocion that needs attention, and it receives a single useful argument: its name.

func (r *PromocionReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    var promo Promocion
    if err := r.Get(ctx, req.NamespacedName, &promo); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }
    // ... create the ConfigMap, Deployment and Service ...
    // ... update the status ...
    return ctrl.Result{}, nil
}

Questions and answers

Reconcile only receives a name. Shouldn't it receive what changed?

And here is the most important idea in the whole module: no, and it is deliberate.

A controller never receives an "event" like "Deployment X was deleted". It receives a name and has to find out for itself how the world looks. This is called level-based reconciliation, as opposed to reacting to events (edge-based), and it has a huge consequence:

If your controller misses an event, nothing happens. If it restarts, nothing happens. If it falls asleep for an hour, nothing happens. If it starts in a cluster that already has half the work done, nothing happens.

An event-based controller, on the other hand, goes out of sync forever as soon as it loses one message. And messages do get lost: the process restarts, the connection drops, a watch expires.

It is the deep reason why Kubernetes is reliable. And it is a principle you can steal for your own systems.

Why client.IgnoreNotFound(err) when the Promocion does not exist?

Because the Promocion has already been deleted, and there is nothing to clean up. The objects it created will disappear on their own, through their ownerReferences. Returning an error here would only make controller-runtime retry forever to reconcile an object that no longer exists.

What does controllerutil.CreateOrUpdate do?

It is the kubectl apply of code, and it contains the "compare" of the loop:

controllerutil.CreateOrUpdate(ctx, r.Client, dep, func() error {
    dep.Spec.Replicas = &replicas
    // ...the desired state...
    return controllerutil.SetControllerReference(&promo, dep, r.Scheme)
})

It does a Get; if the object does not exist, it creates it; if it exists, it applies your mutating function and, only if something has really changed, does an Update.

Notice what this means: you describe the desired state the same way every time, whether it already exists or not. You do not write an if exists { update } else { create }. That is what makes your Reconcile idempotent: you can call it a thousand times in a row and the result is the same as calling it once.

And SetControllerReference?

It is the line that stamps the ownerReferences onto the child object:

ownerReferences:
- apiVersion: tienda.example.com/v1
  kind: Promocion
  name: rebajas-verano
  uid: 8f3e...          # <- the UID, not just the name
  controller: true
  blockOwnerDeletion: true

It tells Kubernetes: "this Deployment belongs to that Promocion". From then on, the cluster's garbage collector takes care of the cleanup, and your operator does not write a single line of code to delete anything.

The uid is a subtle and necessary detail: the name is not enough. If someone deletes rebajas-verano and creates another Promocion with the same name, it is a different object, with another UID, and the orphans of the first one must not be adopted by mistake.

Why does the operator write the status with r.Status().Update() and not with a normal Update()?

Because status is a subresource: it lives at its own endpoint (/promociones/rebajas-verano/status) and has its own RBAC permission. And that separation is intentional:

The spec belongs to the user. The status belongs to the operator.

The user declares what they want; the operator reports how it is going. If your operator rewrites the spec of its own resource, you have built a machine for fighting with your team's GitOps: Argo applies the spec from the repository, the operator changes it, Argo applies it again, and so on until the end of time.

That is why the ClusterRole you will apply later grants get/list/watch on promociones, but only update on promociones/status. The API forces you to design well.

SetupWithManager: where the real difference is

These five lines are what separate this operator from a script in a loop:

func (r *PromocionReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&Promocion{}).            // the object we govern
        Owns(&appsv1.Deployment{}). // ...and the ones that belong to it
        Owns(&corev1.Service{}).
        Owns(&corev1.ConfigMap{}).
        Named("promocion").
        Complete(r)
}

For(&Promocion{}) is the obvious part: reconcile when a Promocion changes.

Owns(&appsv1.Deployment{}) is the part that is not obvious, and it is the best thing in controller-runtime.

It tells the manager: "watch the Deployments too. When one changes, look at its ownerReferences, and if it belongs to a Promocion, reconcile that Promocion".

In plain terms: if someone deletes the rebajas-verano Deployment, controller-runtime detects it instantly (through an API server watch, not a sleep), works out that its owner is the Promocion rebajas-verano, and queues a reconciliation of that Promocion. Your Reconcile runs in milliseconds, finds that the Deployment is missing, and recreates it.

A script in a loop with sleep 5 would do the same... five seconds later, and burning API server CPU with a full list every five seconds even if nothing has changed. With Owns() there is no polling: there is a watch, and the API server pushes the change to you.

This is exactly what every native Kubernetes controller does.

And where are the HTTP client, the cache, the work queue, the retries?

The manager provides them, and you do not see them. mgr.GetClient() gives you back a client that reads from an in-memory cache fed by watches (that is why a Get inside a Reconcile does not hit the API server) and writes directly against the API.

And if your Reconcile returns an error, controller-runtime requeues the object with exponential backoff, by itself. There is no retry to write.

return ctrl.Result{}, err                                    // retry with backoff
return ctrl.Result{}, nil                                    // done
return ctrl.Result{RequeueAfter: 30 * time.Second}, nil      // come back in 30s

Those three return values are the entire flow-control API of a controller. Everything else is your business logic.

Continue with the next unit to run it.

An operator is not a Kubernetes component. It is an ordinary program that talks to the API server. And the best way to internalize that is to run it from your own machine, before putting it in any container.

Step 1: go run .

cd /home/laborant/operador

go run .

Leave it running. The first thing that shows up is this:

INFO    operador-promociones arrancando
INFO    controller-runtime.metrics  Starting metrics server
INFO    Starting EventSource    {"controller": "promocion", "source": "kind source: *main.Promocion"}
INFO    Starting EventSource    {"controller": "promocion", "source": "kind source: *v1.Deployment"}
INFO    Starting Controller {"controller": "promocion"}
INFO    Starting workers    {"controller": "promocion", "worker count": 1}

(The first line, arrancando, is Spanish for "starting".) Those four EventSource are the four calls in SetupWithManager: the For(&Promocion{}) and the three Owns(...). And as soon as the workers start, the first reconciliation of rebajas-verano arrives.

If the terminal goes quiet for a few seconds, it is not hung: it is compiling. The first go run . links the whole of controller-runtime and prints nothing while it does. If after half a minute there is still not a single line, then something is indeed off, and the place to look is go build ., which will tell you what fails without starting anything.

Open a second terminal (the dev-machine tab allows several) and look at the cluster:

kubectl get all

kubectl get configmap rebajas-verano-html

There they are: a Deployment with three replicas, a Service and a ConfigMap. Nobody applied them. They exist because a Promocion object says they must exist, and there is a program (in your terminal, not in the cluster) that takes it seriously.

Which credentials is it using?

Yours. Look at this line in main.go:

mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{Scheme: scheme})

GetConfigOrDie() looks for credentials in this order:

1. The ServiceAccount mounted in the Pod, if the process runs inside the cluster (/var/run/secrets/kubernetes.io/serviceaccount/).
2. $KUBECONFIG or ~/.kube/config, if it runs outside.

The same binary serves for both, without changing a line. Right now it is using your kubeconfig, which is an administrator's. That is why it works without you having granted it a single permission.

Remember that, because in ten minutes it is going to stop working.

Step 2: The status

Look at the Promocion:

kubectl get promociones

NAME             BANNER                       DESEADAS   LISTAS   FASE    EDAD
rebajas-verano   https://tienda.example.com   3          3        Listo   2m

The column names are the Spanish ones from the CRD (DESEADAS is desired, LISTAS is ready, FASE is phase, EDAD is age, and Listo means ready). The LISTAS and FASE columns do not come from the spec: the operator wrote them, in the status. It is the channel through which an operator tells the user how things are going.

kubectl get promocion rebajas-verano -o jsonpath='{.status}' | jq .

This is exactly what a Deployment does when kubectl get deployments shows you 3/3. Your resource already behaves like a native one.

Step 3: Break it, with the logs in front of you

This is the moment of the lesson. Do not close the terminal where go run . is running. Put it where you can see it, and from the other one:

kubectl delete deployment rebajas-verano

Look at the operator's logs. The reconciliation fires immediately. Not in five seconds: in the very same instant.

And now the question: your controller only declares For(&Promocion{}). The Promocion has not changed. Why was it reconciled?

Because of this:

Owns(&appsv1.Deployment{}).

Controller-runtime is watching the Deployments. When yours disappeared, it looked at its ownerReferences, saw that it belonged to the Promocion rebajas-verano, and queued a reconciliation of that Promocion. Your Reconcile ran, found the Deployment missing, and recreated it.

This is the real Kubernetes mechanism, and there is no polling anywhere: there are watches, and the API server pushes the changes.

Try scaling it by hand too:

kubectl scale deployment rebajas-verano --replicas=1

It goes back to 3. The Promocion says 3. Your kubectl is not a final order: it is a temporary discrepancy with the desired state, and the reconciliation corrects it.

Step 4: Now, inside the cluster

Stop the go run . with Ctrl+C.

An operator has to live inside the cluster: with high availability, with its own identity, and without depending on anyone's laptop. Time to package it.

Look at the Dockerfile:

FROM golang:1.26-alpine AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY *.go ./
RUN CGO_ENABLED=0 go build -o /operador .

FROM gcr.io/distroless/static:nonroot
COPY --from=build /operador /operador
USER 65532:65532
ENTRYPOINT ["/operador"]

Why two stages? Because compiling needs the whole Go toolchain (about 800 MB) and running needs nothing. Go compiles a static binary: no interpreter, no shared libraries, no dependencies.

The final image is distroless/static: no shell, no package manager, not even ls. Just your binary. It weighs about 50 MB and its attack surface is practically nil. An attacker who gets code execution inside that container has nothing to even look around with.

Build the image. Notice the name: it carries the registry host up front, and that is not decoration: it is what tells Docker where to publish it.

cd /home/laborant/operador

docker build -t registry.iximiuz.com/tienda/operador-promociones:v1 .

docker images registry.iximiuz.com/tienda/operador-promociones

There it is, in the Docker store on dev-machine. And there it is no use to anyone, because the one that has to pull it is not Docker: it is the kubelet on each node. dev-machine is not a cluster node: it is your work machine, and the cluster is cplane-01, node-01 and node-02. Publish it to the playground's registry, which they all can see:

docker push registry.iximiuz.com/tienda/operador-promociones:v1

Why isn't building it enough? Because it is the classic stumble of everyone starting out: the image "exists" (docker images lists it) and the Pod sits in ImagePullBackOff. The kubelet does not look at your local Docker. Never. Not in this playground and not in production.

A registry is the normal answer to that problem, and it is the one you will always use: the registry is the meeting point between whoever builds and whoever runs. That is why deploy.yaml references the image by its full name with host, and why it carries an imagePullSecrets: this playground's registry asks for credentials, and the kubelet needs its own.

Deploy:

The operator's manifests are in manifiestos/, and they are applied in the order they are explained. First the identity:

kubectl apply -f manifiestos/sa.yaml

A ServiceAccount and nothing else: six lines, not a single permission. It has just been born and cannot do absolutely anything, and that is exactly what you want to check in a moment.

Then the workload, which uses it:

kubectl apply -f manifiestos/deploy.yaml

kubectl get pods -l app=operador-promociones

Step 5: The operator is no longer an administrator

kubectl logs -l app=operador-promociones --tail=30

The manager starts, tries to sync its cache... and the API server slams the door in its face. It cannot even do a list of Promociones.

And it is quite right. Before, the operator used your administrator kubeconfig. Now it uses the operador-promociones ServiceAccount, which has exactly zero permissions.

An operator is one more client of the API server, and the API server makes exceptions for no one.

Step 6: The permissions, and only the necessary ones

Open rbac.yaml before applying it:

cat /home/laborant/operador/manifiestos/rbac.yaml

rules:
# Observe the Promocion objects. READ ONLY: the spec belongs to the user.
- apiGroups: ["tienda.example.com"]
  resources: ["promociones"]
  verbs: ["get", "list", "watch"]
# Write the status. It is a SEPARATE subresource, with its own permission.
- apiGroups: ["tienda.example.com"]
  resources: ["promociones/status"]
  verbs: ["get", "update", "patch"]
# Act: the derived objects.
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: [""]
  resources: ["services", "configmaps"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

Notice the first two rules. On promociones, the operator is read-only. On promociones/status, it can write. That is the contract we were talking about: the spec belongs to the user, the status belongs to the operator, and RBAC really enforces it. It is not a convention: it is a permission.

Why also list and watch on Deployments, Services and ConfigMaps, if I only create them? Because your controller declares Owns() on all three, and Owns() opens a watch. Without watch permission, the manager does not even start.

Apply it:

kubectl apply -f /home/laborant/operador/manifiestos/rbac.yaml

kubectl logs -l app=operador-promociones -f

The logs change instantly. The operator reconciles.

And check the permissions by impersonating it:

kubectl auth can-i list promociones.tienda.example.com \
  --as=system:serviceaccount:default:operador-promociones -A

kubectl auth can-i update promociones.tienda.example.com --subresource=status \
  --as=system:serviceaccount:default:operador-promociones -A

kubectl auth can-i delete secrets \
  --as=system:serviceaccount:default:operador-promociones -A

The first two say yes. The third says no, and that is a victory too: if someone compromises your operator, they do not walk off with the cluster's Secrets.

kubectl auth can-i VERB [TYPE | TYPE/NAME | NONRESOURCEURL]

The operator lives inside the cluster, with its identity and its permissions. Now let's put it to the test.

Step 7: Try to break it (again, but for real)

Keep the logs in front of you:

kubectl logs -l app=operador-promociones -f

And in another terminal, sabotage:

kubectl delete deployment rebajas-verano
kubectl delete service rebajas-verano
kubectl delete configmap rebajas-verano-html

All three come back. Instantly.

Delete the Deployment twenty times if you like. It comes back twenty times.

This is what it means for Kubernetes to be declarative, and it is worth spelling out because it changes the way you operate systems:

You do not tell the cluster what to do. You tell it how the world should be. And there is someone in there, checking it without rest.

Your kubectl delete was not a final order. It was a temporary discrepancy with the desired state, and the reconciliation corrected it. Exactly the same thing that happens to a Pod you delete from a ReplicaSet.

Step 8: The Custom Resource is the source of truth

The Deployment is not in charge. The Promocion is. Prove it:

kubectl patch promocion rebajas-verano --type=merge -p '{"spec":{"replicas":5}}'
kubectl get promociones -w

Look at the columns: DESEADAS jumps to 5 instantly, FASE changes to Desplegando (deploying), and LISTAS climbs up to 5, at which point FASE goes back to Listo.

Notice what has just happened inside, because they are two different reconciliations:

1. You changed the Promocion → For(&Promocion{}) detected it → Reconcile set the Deployment to 5 replicas.
2. The Deployment changed (its ready replicas went from 3 to 5) → Owns(&appsv1.Deployment{}) detected it → Reconcile ran again → it updated the Promocion's status.

Without Owns(), the status would have stayed frozen at 3 until the next time someone touched the Promocion. With Owns(), the status follows the real world.

You have scaled an application by modifying an object that did not exist half an hour ago and that you defined yourself. That is exactly what the PostgreSQL, Kafka or Prometheus operators do: they give you a high-level object (kind: PostgresCluster) and translate your intent into dozens of low-level resources.

Step 9: Delete everything with a single line

Delete the Promocion. Only the Promocion:

kubectl delete promocion rebajas-verano

kubectl get all

kubectl get configmap

The Deployment, the Service, the ConfigMap and the five Pods: everything is gone.

And the most interesting part: your operator does not have a single line of cleanup code. Look for it in main.go. It is not there. Not one Delete. Not one defer.

The Kubernetes garbage collector did it, following the ownerReferences that SetControllerReference stamped on each object when creating it. The same ownerReferences by which deleting a Deployment deletes its ReplicaSets, and deleting a ReplicaSet deletes its Pods.

Look at the operator's logs: you will see one last reconciliation in which r.Get() returns NotFound, and that line you now fully understand:

return ctrl.Result{}, client.IgnoreNotFound(err)

"The object no longer exists. There is nothing to do." And there is not.

Why this operator is not fit for production

Your operator works, and 90% of the operators in the world do nothing conceptually different. But to put it in charge of something important it is missing four things, and it helps to know their names:

1. Leader election. Right now it runs with one replica. If you ran two, they would fight: each would overwrite the other on every pass. Serious operators run several replicas for availability, but only one reconciles; the others wait their turn through a Lease object, the same mechanism the kube-scheduler and the kube-controller-manager use. In controller-runtime it is a manager option:

ctrl.Options{LeaderElection: true, LeaderElectionID: "operador-promociones"}

2. Finalizers. The garbage collector cleans up what is inside the cluster. It knows nothing about an S3 bucket, a managed database or a DNS entry. That is what finalizers are for: a marker in metadata.finalizers that prevents the object from being fully deleted until the operator has done the external cleanup and removed its marker.

They are also the number one cause of objects stuck in Terminating forever: a finalizer whose operator no longer exists, and which therefore nobody is ever going to remove. If you ever run into one, now you know what to look at (and why kubectl patch ... -p '{"metadata":{"finalizers":null}}' is an operation you have to understand before running it).

3. Events. A serious operator emits Events on the objects it manages, so that kubectl describe promocion rebajas-verano tells its story. It is one line with the manager's EventRecorder, and it is the difference between an operator you can debug and one you cannot.

4. Strong validation. Your openAPIV3Schema already validates types and ranges. For richer rules ("replicas cannot go down if spec.protected is true") there are CEL validation rules inside the CRD itself, and admission webhooks. Always in this order: the schema is free, CEL is cheap, a webhook is one more service to maintain and one that can take down your cluster if it goes down.

Summary

• An operator is a controller for a custom resource. And a controller is a loop: observe, compare, act.
• Reconciliation is level-based, not event-based: Reconcile receives a name, not a "what changed". That is why it can restart, miss events or start halfway through, and still converge.
• CreateOrUpdate makes the "act" idempotent: you describe the desired state the same way every time, whether it already exists or not.
• Owns() is what separates an operator from a script with a sleep: it opens a watch on the child objects and reconciles the parent when they change. No polling, and in milliseconds.
• The spec belongs to the user, the status belongs to the operator. They are different subresources, with different RBAC permissions, and the API forces you to respect it.
• SetControllerReference connects each child to its parent. The garbage collector does the cleanup, and your code does not write a single Delete.
• An operator is one more client of the API server: the same binary inside and outside the cluster, and without RBAC it does nothing.
• What it lacks for production has concrete names: leader election, finalizers, events and validation. Now you know which they are and why.
• The Prometheus Operator, cert-manager and the PostgreSQL operator do exactly this. They are no longer black boxes.