This course was created by a community author. Community content is reviewed by the iximiuz Labs team on a best effort basis.

Course Last updated: Jul 23, 2026

Kubelings — Learn Kubernetes the Rustlings Way

by Madhan Kumaravelu · on

Kubernetes

Learn Kubernetes by fixing small, broken-on-purpose clusters — one scenario at a time — until an automated check turns green. Each lesson drops you into a live multi-node cluster with a realistic fault to diagnose and repair: rolling updates, DaemonSets, StatefulSets, Jobs & CronJobs, autoscaling, and resource limits. Hands-on, check-driven, CKA/CKAD-aligned.

#cka #ckad #workloads #hands-on

Kubelings — Learn Kubernetes the Rustlings Way (cover image)

Reset Progress

Are you sure you want to reset your progress for this course? This action cannot be undone.

About This Course

Kubelings is hands-on Kubernetes practice in the spirit of rustlings and golings: a sequence of broken or empty clusters, each with an automated check that only passes when you've genuinely fixed the problem.

Every lesson runs in its own live, multi-node cluster — no setup, no cleanup. Read the situation, fix the cluster with kubectl, and watch the check go green.

Work the modules top to bottom: Foundations → Workloads → Config & Storage → Networking → Scheduling → Security → Internals → Observability & SRE → War Stories (capstone replays of real, cited production postmortems — the Zalando ndots DNS outage is already a runnable lesson in Networking) → Platform Engineering → Node & Control Plane — host-level break/fix on real machines (kubelet, containerd, etcd, the API server), the failures that live below kubectl.

Most scenarios also run locally on kind; the host-level labs are marked iximiuz Labs only, since they need a real node with systemd and a kernel. See the project repo for the local runner and the full incident library.

What's Inside?

Module 1: FoundationsHow Kubelings works, plus the Kubernetes bedrock: pods, Deployments, Services, namespaces, labels & selectors, and the kubectl habits you'll lean on in every later module. Each scenario is a small broken cluster with an automated check.
Lesson: Welcome to KubelingsHow the course works: read the situation, fix the live cluster, watch the automated check turn green.
Lesson: CrashLoopBackOff: read the logs before you guessThe new orders service won't stay up — pods flash into Running, die within a second, and settle into CrashLoopBackOff. Nothing is wrong with the image. Learn the triage loop every later lesson builds on: describe → logs → fix → watch.
Lesson: Expose a Deployment: your first ServiceThe web frontend is running — two healthy nginx pods — but nothing in the cluster can reach it: there's no Service. Create one, watch endpoints populate, and learn how label selectors stitch traffic to pods.
Lesson: The Service that routes to nothingThe api Service exists, DNS resolves, the pods are healthy — and every request times out. Its endpoints list is empty: the selector matches zero pods. One character is wrong somewhere. Find it.
Lesson: Namespaces: walls, names, and crossing themThe platform team wants a staging environment isolated from everything else, and the app config copied into it. Learn what namespaces isolate (names, RBAC, quotas) and what they don't (network, nodes) — and how DNS crosses the wall.
Lesson: ImagePullBackOff: the tag that never existedA hotfix deploy is stuck — pods sit in ImagePullBackOff and the rollout never completes. The registry is fine. The image is fine. The tag is not. Learn to read pull errors and roll the Deployment back to health.
Lesson: kubectl detective: find the broken oneFive microservices deployed, dashboards say "one of them is down", and that's all anyone knows. No names, no hints. Use get/describe/events to find the odd one out and bring it back. Speed comes from method, not luck.
Module 2: WorkloadsCheck-driven scenarios covering the CKA/CKAD Workloads domain: rolling updates, DaemonSets, StatefulSets, Jobs, CronJobs, HorizontalPodAutoscaler, right-sizing resources to stop OOMKills — and, coming up, probes, init containers, PDBs, and QoS classes.
Lesson: Fix the Rolling Update: unsafe maxSurge/maxUnavailableA web Deployment ships every release with a full outage — its rollout strategy tears down all pods at once and never surges new ones. Fix the rolling update strategy so deploys stay available, then confirm the Deployment is healthy.
Lesson: Build a Node-Level Log Collector DaemonSetOps needs a log shipper running on every node — exactly one pod per node, now and on any node added later. Build a DaemonSet that satisfies this and confirm it lands a Ready pod on each node.
Lesson: StatefulSet with Stable Pod Identity + Headless ServiceA stateful app needs stable, predictable pod names and per-pod DNS. Build a StatefulSet fronted by a headless Service so each replica gets a durable network identity, then confirm all replicas are Ready.
Lesson: The Job That Never FinishesA one-shot data-import Job runs forever and never reports completion, so the pipeline that waits on it is stuck. Diagnose why, then make the Job actually complete.
Lesson: CronJob Pileup: fix the concurrencyPolicyA every-minute CronJob runs longer than a minute, so overlapping runs stack up and exhaust resources. Configure the CronJob so a new run never starts while the previous one is still going.
Lesson: Autoscale a Deployment with an HPA (1 → 5)A CPU-bound web app needs to scale out under load and back in when idle. With metrics-server already running, create a HorizontalPodAutoscaler that scales the Deployment between 1 and 5 replicas on CPU.
Lesson: OOMKilled CrashLoop: right-size the memory limitA cache Deployment is stuck in CrashLoopBackOff — every pod is OOMKilled seconds after start. Diagnose the resource misconfiguration and give the workload enough memory to run.
Lesson: The liveness probe that kills healthy podsSomeone "added health checks" to the payments service and now it restarts forever — the liveness probe checks the wrong port, so the kubelet executes a perfectly healthy container every few seconds. Learn what liveness and readiness actually promise, and when each one should fail.
Lesson: Slow starter vs impatient livenessA legacy service needs ~40 seconds to warm up. Its liveness probe gives it 10. The pod is killed mid-boot, restarts, gets killed again — it will literally never finish starting. Fix it the modern way: a startupProbe that holds liveness fire until boot completes.
Lesson: Stuck at Init: the gate that never opensThe reports pod has been Init:0/1 for twenty minutes. Its init container is waiting for configuration that will never arrive — it's checking the wrong name. Learn how init containers gate pod startup, and how to debug the gate.
Lesson: The drain that never finishesNode maintenance tonight. But kubectl drain has been "evicting" the same two pods for ten minutes — a PodDisruptionBudget demands more availability than the deployment has replicas. Learn the PDB math and fix it so maintenance and availability can both be true.
Lesson: QoS classes: who gets killed firstThree tiers of workload, one node under memory pressure — and the kernel needs someone to die. Kubernetes decides the order via QoS classes computed from your requests/limits. Make the payment service Guaranteed, the batch job honest, and learn to read .status.qosClass before the OOM killer reads it for you.
Lesson: Blue/green: the deploy you can undo in one secondRolling updates replace pods gradually — great until a bad release must be yanked *now*. Blue/green runs old and new side by side and flips traffic with one selector change. The green stack is up, tested, and receiving zero traffic; flip the switch without dropping a request, and keep the way back.
Lesson: CPU throttling: the latency tax nobody seesThe service is "healthy" — CPU usage looks modest, no restarts, no errors — and p95 latency is garbage. The kernel is pausing your process dozens of times a second to enforce a CPU limit. Read the throttling counters the kernel keeps inside every container, then fix the limit like Omio and Buffer had to.
Lesson: No shell, no exec, no problemA distroless production pod is misbehaving and kubectl exec greets you with "no such file or directory" — there is no shell in the image. Attach an ephemeral debug container with kubectl debug, read the app's filesystem through the shared PID namespace, and fix the bad config it's choking on.
Lesson: Three pods, three broken patternsSidecar, ambassador, adapter — the three classic multi-container patterns, each deployed here with one bug: a log-forwarder tailing the wrong path, a proxy pointed at the wrong port, an adapter writing to the wrong place. Fix all three and prove each pattern actually does its job.
Lesson: Drill — the readiness probe that flapsSynthetic drill of a failure pattern reported across many production clusters: a service throws intermittent 502s because its pods cycle in and out of the endpoint pool every few seconds. The readiness probe is wired to a hair trigger — one blip, one second, and the pod is yanked. Tune the probe so it tolerates blips without lying about health.
Lesson: Drill — zombie CronJobs pile upSynthetic drill of a failure pattern reported across many production clusters: a CronJob has been running for weeks with history limits of 50, and dozens of completed Jobs and their pods now clutter the namespace, bloat every LIST, and slow the API server's watchers. Clear the pile and fix the limits so it never regrows.
Lesson: Drill — the rolling update that deadlocksSynthetic drill of a failure pattern reported across many production clusters: a rollout stuck for hours at "1 old replicas are pending termination". maxUnavailable: 0 means no old pod may die until a new one is Ready — and the new one can never be Ready, because its resource request fits on no node. Nothing crashes, nothing progresses. Break the deadlock.
Lesson: VPA: the recommender that watched the wrong appRight-sizing by hand (like the oomkill lesson) doesn't scale past a few services — the Vertical Pod Autoscaler watches real usage and recommends requests. This cluster's VPA has watched for days and recommended nothing: its targetRef names a Deployment that doesn't exist. Point it at the real workload and read your first recommendation.
Lesson: KEDA: the ScaledObject that never scaledNightly report traffic needs 3 replicas during business hours; HPA can't scale on "what time it is" — KEDA can. But this ScaledObject sits Ready=False and the deployment never leaves 1 replica: the cron trigger's schedule doesn't parse. Fix the trigger, watch KEDA drive the HPA it manages for you.
Lesson: Incident replay — restartPolicy: Never did not mean never (Universe)Replay of Universe's cited 2017 outage: a reindex Job was written with restartPolicy: Never in the belief that it would run once. That field belongs to the *pod* — the Job controller kept making new ones, forever, and the greedy retries took the nodes and the cluster with them. Bound the failure without fixing the crash.
Module 3: Config & StorageWiring configuration and state into workloads: ConfigMaps and Secrets (env vs mounts, hot-reload gotchas), PersistentVolumes and claims, StorageClasses, access modes, and the failure modes around volumes that refuse to bind or die.
Lesson: CreateContainerConfigError: the key that isn't thereThe pricing service won't start — not crashing, not pulling, just CreateContainerConfigError. It wants an env var from a ConfigMap key that doesn't exist. Learn how config wires into containers (env vs mounts) and why the two behave differently when config changes.
Lesson: ContainerCreating forever: the Secret that isn'tThe gateway pod has been ContainerCreating for ten minutes. Its TLS Secret volume references a Secret that doesn't exist under that name. Learn how Secret volumes gate container creation, and the difference between a missing Secret and a missing key.
Lesson: PVC Pending: a claim nobody answersThe analytics database needs a disk. Its PersistentVolumeClaim has been Pending for half an hour — it names a StorageClass that doesn't exist in this cluster. Learn the PV/PVC/StorageClass triangle and how dynamic provisioning actually answers a claim.
Lesson: One disk, two nodes: the access-mode trapThe wiki scaled to two replicas for "high availability" — and the second pod has been Pending ever since. Both want the same ReadWriteOnce volume, and anti-affinity pushes them to different nodes. Learn what access modes really promise and why RWO + spread = deadlock.
Lesson: Drill — the PVC stuck TerminatingSynthetic drill of a failure pattern reported across many production clusters: a teardown deletes a PersistentVolumeClaim, it goes Terminating… and stays there. Forever. A finalizer is protecting it from a consumer nobody remembers. Find the ghost, release the claim — without the corrupting shortcut.
Lesson: Kustomize: one base, many environments, zero driftStaging and prod were "the same YAML, copy-pasted" — until someone hotfixed prod by hand and the copies disagreed. Rebuild the setup as a kustomize base plus a prod overlay with kubectl's built-in -k, and make the drift impossible to express.
Lesson: Helm: history, rollback, and the failed releaseA Helm upgrade shipped a bad image tag and the release is now failed — pods stuck in ImagePullBackOff. Read the release history, roll back to the last good revision, then re-ship the intended change with corrected values. helm install/upgrade/rollback, the full loop.
Lesson: Drill — ghost endpoints after scale-downSynthetic drill of a failure pattern reported across many production clusters: every scale-down or rollout of a service throws a burst of 503s. terminationGracePeriodSeconds is 0 and there is no preStop hook — pods die the instant deletion starts, while their IPs are still in the endpoint list. Give termination the time the network needs.
Lesson: Drill — the Secret that was rotated but never reloadedSynthetic drill of a failure pattern reported across many production clusters: security rotated a database password, the Secret shows the new value, the app still authenticates with the old one — because the pod got it as an env var at startup, and env vars never reload. Move the injection to a volume mount and pick up the rotation.
Lesson: Drill — the namespace stuck TerminatingSynthetic drill of a failure pattern reported across many production clusters: a namespace has been Terminating for ten minutes. A custom resource inside it carries a finalizer whose controller was uninstalled long ago — nothing will ever clear it. Find the blocker, and use the finalizer-strip correctly for once: on an object whose owner is permanently gone.
Lesson: Databases on Kubernetes: what a StatefulSet does not give youReading — the durability questions Kubernetes does not answer for you: async replication and the failover that loses committed writes, what a PVC really promises, why "the pod restarted somewhere else" is a data question, and when an operator earns its keep. Built on Gravitational's cited PostgreSQL-on-K8s write-up and the operator practice that grew out of it.
Module 4: NetworkingHow traffic actually moves: Service types and endpoints, Ingress, NetworkPolicy, CoreDNS and the infamous ndots amplification, kube-proxy modes — including scenarios reproduced from real production DNS outages.
Lesson: Incident replay — the ndots:5 DNS amplifier (Zalando, Jan 2019)Replay of a real production outage: Zalando's fashion store lost ALL cluster DNS for over an hour when retry traffic, un-cached lookups and Kubernetes' default ndots:5 multiplied every external hostname lookup into ~10 DNS queries, OOMKilling every CoreDNS pod at once. Find the amplifier in this cluster and defuse it.
Lesson: NetworkPolicy blackhole: default-deny ate my trafficSecurity rolled out a default-deny NetworkPolicy — correct move — but nobody wrote the allow rules that were supposed to ship with it. Learn how policies select, combine, and fail silently, and write the allow rule that lets exactly the right traffic through.
Lesson: Connection refused: port vs targetPortEndpoints populated, selector correct, pods healthy — and every request gets "connection refused". The Service forwards to a container port nobody listens on. Learn the port/targetPort/containerPort chain and the difference between refused and timeout.
Lesson: Service types: open a door to the outsideThe demo works inside the cluster, but the client wants to hit it from their laptop. Promote the ClusterIP Service to NodePort on 30080 and understand the ladder: ClusterIP → NodePort → LoadBalancer — what each adds, and what each costs.
Lesson: Incident file — conntrack: the invisible table that fills upRead-through of two real, cited incidents (loveholidays, Preply) where the Linux connection-tracking table — a thing most teams don't know exists — silently dropped production traffic and corrupted DNS. No tasks; this is pattern recognition for a failure you can't reproduce safely on a laptop.
Lesson: Incident replay — 502s on every deploy (Ravelin's endpoint secret)Replay of Ravelin's cited write-up: every rolling update produced a burst of 502s, because pod termination and endpoint removal run in PARALLEL, not in order. The fix is a deliberately "dumb" preStop sleep. Understand the race, then wire the fix.
Lesson: Ingress: three names that must agreeThe new Ingress went out and the site 404s. An Ingress is pure wiring — host to path to service name to service port to pod — and every hop is a name that must match something real. Two of these names don't. Find them the way you found the selector mismatch: follow the chain.
Lesson: Gateway API: routing with a role for everyoneIngress crams infra and app concerns into one object; Gateway API splits them — GatewayClass for the implementation, Gateway for the listener owned by platform, HTTPRoute for the routes owned by app teams. The platform half is already deployed. Write the app team's HTTPRoute and attach it.
Lesson: kube-proxy: there is no proxyReading — what actually happens to a packet addressed to a ClusterIP. The IP that exists on no interface, the iptables chains that rewrite it, the IPVS and nftables alternatives, and where conntrack plugs in. Closes the loop on every Service mystery in this module.
Lesson: CNI: who wires the pod, and what breaks when nobody doesReading — the container network interface from the kubelet's point of view: the conflist file, the plugin binaries, what Calico and Flannel actually install, and the triage tree for pods stuck ContainerCreating — is it the CNI, or is it a NetworkPolicy?
Lesson: kubeconfig: contexts, the merge, and the prod you almost touchedA kubeconfig with prod/staging/dev contexts has current-context pointed at prod — one habitual kubectl apply away from a bad day. Switch to staging, then merge a teammate's kubeconfig into one flattened file without clobbering either. kubectl config end to end.
Lesson: Drill — externalTrafficPolicy: Local blackholes half your nodesSynthetic drill of a failure pattern reported across many production clusters: a Service keeps the client IP with externalTrafficPolicy: Local, but its pods live on one node. Every other node in the load balancer's rotation drops the traffic on the floor — no endpoints, no forwarding, no error anyone can see from inside the cluster. Keep the client IP and make every node answer.
Lesson: Incident file — the cluster that ran out of IP addresses (loveholidays)Read-through of loveholidays' cited GKE incident plus its cousins (Neon, Veracode): pods stuck Pending, the autoscaler refusing to add nodes, and a subnet that was "obviously big enough" — until you count the way Kubernetes counts. No tasks; the arithmetic is done at cluster-creation time and cannot be patched later, so this lesson trains the sizing instinct.
Module 5: Scheduling & PlacementWhere pods land and why: node/pod affinity and anti-affinity, taints and tolerations, topology spread constraints, priority and preemption — and what happens in production when every replica lands on the same failing node.
Lesson: Incident replay — all replicas on the failing node (Moonlight)Replay of Moonlight's cited outage: every pod of the website landed on the same host — the one that then failed. 100% traffic loss with "redundant" replicas. Spread them so one node can never again take the whole service down.
Lesson: Everything Pending: who tainted the nodes?Overnight, every new pod in the cluster stopped scheduling. Someone prepared the workers for a "dedicated batch pool" migration that was cancelled — but the taints stayed. Learn how taints repel, how tolerations permit, and how to read the scheduler's rejection notes.
Lesson: Topology spread: balance, not just separationSix replicas of the session store — five on one node, one on another. Anti-affinity can't fix this: it only says "not together", not "evenly". Learn topologySpreadConstraints, maxSkew math, and when soft beats hard.
Lesson: Incident replay — the priority that ate production (Grafana Labs)Replay of Grafana Labs' cited outage: introducing PriorityClasses with the wrong defaults caused Kubernetes to PREEMPT — evict — production pods to make room. Set the priority tiers right so the important thing wins the next resource fight.
Lesson: Drill — the noisy neighborSynthetic drill of the fleet-wide classic: one workload declares no resources and feasts, its neighbor is CPU-throttled into 2-second latency. Nothing crashes; everything is slow. Diagnose with resource math, fix with honest declarations.
Module 6: SecurityLocking the platform down: RBAC and ServiceAccounts (and how over-broad grants become cryptominers), Pod Security standards, admission webhooks and the ways a single misconfigured webhook can take the whole cluster write-path down.
Lesson: cluster-admin for a bot: scope it downThe CI bot needs to read pods in one namespace. Someone gave it cluster-admin — write access to everything, everywhere, forever. Replace the god-grant with a least-privilege Role and prove the difference with kubectl auth can-i.
Lesson: Incident replay — the exposed dashboard (JW Player's cryptominer)Replay of JW Player's cited incident: an internal ops tool got a public LoadBalancer with no auth, and attackers used it to launch a cryptominer across the cluster. Find the pod nobody deployed, kill it, and shut the door it walked through.
Lesson: Incident replay — the webhook that froze the cluster (Jetstack)Replay of Jetstack's cited GKE outage: a validating webhook with failurePolicy=Fail lost its backing pods, so the API server rejected EVERY write — including the ones needed to recover. Fix the failure policy and scope so a webhook can never again take the whole cluster hostage.
Lesson: Pod Security: the privileged pod that shouldn't existA vendor Helm chart wants a privileged, host-network pod with hostPath to the Docker socket — a full container escape, one YAML away from root on the node. Enforce Pod Security Standards on the namespace and rework the pod to run unprivileged.
Lesson: The token in every pod (and who's using yours)Every pod in the namespace is carrying an API credential nobody asked for — the default ServiceAccount token, automounted since day one. Give the one workload that needs API access its own identity, and take the credential away from everything that doesn't.
Lesson: Hardening: take away everything the workload doesn't useThe worker runs as root with a writable filesystem, every Linux capability, and no syscall filter — and uses none of it. Strip it to what it actually needs: non-root, read-only root filesystem, zero capabilities, RuntimeDefault seccomp. The pod should barely notice; an attacker inside it definitely will.
Lesson: Audit the cluster: CIS benchmark with kube-benchYou hardened pods; now audit the platform itself. Run kube-bench — the CIS Kubernetes Benchmark as a Job — against this cluster, read its PASS/FAIL/WARN verdicts on the API server, kubelet, and etcd configuration, and learn which findings matter before an auditor (or an attacker) reads them to you.
Lesson: The hardening you can't kubectl: control plane, supply chain, runtimeReading — the CKS-shaped half of security that lives outside the pod spec: encrypting Secrets at rest, audit logging, API server flags, image supply chain (digests, scanning, signing), and runtime security (Falco, gVisor, AppArmor). Where each control lives, what it stops, and who owns it.
Lesson: Egress lockdown: the miner needs a phone lineThe cryptominer from lesson 6.2 did its damage over outbound connections — and outbound is the direction nobody polices. Lock the payments namespace down: default-deny all egress, then allow back exactly two flows — DNS, and the payment gateway. Everything else, including the next miner, dials a dead line.
Lesson: Tags lie, digests don't: pin the supply chainA tag like nginx:1.27-alpine is a mutable pointer — whoever controls the registry controls what it points at, and yesterday's audited image may not be tomorrow's. Find the digest of what checkout is actually running and pin the deployment to it, making the image content-addressed and unswappable.
Lesson: Gatekeeper: the policy that rejected nothingOPA Gatekeeper is installed and a Constraint says "no :latest tags" — yet nginx:latest sails right in. The Rego in the ConstraintTemplate reads a field that doesn't exist on a Pod, so the violation never fires. Fix the policy so it actually denies, without denying everything else.
Lesson: Kyverno: the policy that blocked kube-systemA Kyverno ClusterPolicy in Enforce mode requires an app label on every pod — in every namespace, every team, cluster-wide. Only Kyverno's default resourceFilters kept it from bricking kube-system too. Scope the policy to where it belongs, add a mutation that fixes pods instead of rejecting them, and keep one hard deny.
Lesson: Scan it, then pin it: trivy and the digestA Deployment is running nginx:1.14 — an image with a museum's worth of known CVEs — referenced by a mutable tag on top. Scan it with trivy, find a clean replacement, and pin the fix by digest so nobody can swap the bytes behind your back.
Lesson: Signatures and SBOMs: trust, but verify with cosignReading with a hands-on core — what image signing actually proves, how Sigstore keyless signatures work, and what an SBOM buys you. Then do it for real: cosign-verify a publicly signed image and record the digest the signature vouches for.
Lesson: seccomp on, AppArmor understoodA pod is running with an unfiltered syscall surface — all ~350 syscalls exposed to whatever escapes the app. Turn on the runtime's default seccomp profile the Kubernetes way, then read the AppArmor runbook for the host-side twin you'll meet on real nodes and the CKS.
Lesson: Encryption at rest: the full runbookEncrypt Secrets at rest on a real control plane: write an EncryptionConfiguration, wire the kube-apiserver static pod, and prove ciphertext with etcdctl — then run the re-encrypt migration everyone forgets. Covers provider order semantics, aescbc vs KMS v2, and key rotation without locking yourself out.
Lesson: Audit policy: who touched that Secret?Turn on audit logging for real: write a policy that captures Secret reads and pod exec without drowning in kubelet noise, wire the kube-apiserver static pod, then answer "who read that Secret?" from the log — without copying the Secret itself into it.
Lesson: Falco: alarms for the attack you didn't preventRuntime detection with Falco: how the eBPF probe sees every syscall, the rules language, writing the shell-in-a-container rule, and the install runbook. Then run it for real — install the DaemonSet on a live node, spawn an interactive shell in a container, and prove your rule actually fired an alert (and that the tty-less exec everyone tries first does not).
Lesson: Incident file — the identity sidecar that added a zero to latency (Adevinta)Read-through of Adevinta's cited migration incident: moving to Kubernetes made p99 latency 10× worse, and the culprit was the machinery that gives pods cloud credentials — a metadata-intercepting agent in the path of every AWS call, amplified by DNS. No tasks; the failure needs a cloud IAM plane, so this lesson trains how pod identity actually works and where it hurts.
Module 7: Internals — how Kubernetes actually worksUnder the hood: the API server request & admission flow, etcd as the source of truth, the scheduler's filter/score loop, controller-manager reconciliation, kubelet ↔ CRI, CNI, informers & watches, and leader election. You'll poke each component live on a kind control plane.
Lesson: The reconcile loop: why deleted pods come backDelete a pod and it reappears in seconds — not magic, but a controller comparing desired state to actual and acting on the difference. Watch the ReplicaSet controller heal a Deployment in real time, and learn the single idea the whole control plane is built on.
Lesson: What the scheduler actually does (bypass it to find out)The scheduler feels like a black box. It isn't — its entire job is to write one field: spec.nodeName. Prove it by scheduling a pod yourself, bypassing the scheduler entirely, then reason about filter-and-score placement.
Lesson: etcd: the one source of truthEvery object you have ever created lives as a key in etcd. Write a ConfigMap, then find it byte-for-byte inside etcd at /registry/... — and understand why "the API server is stateless" and "back up etcd or lose everything" are the same fact.
Lesson: The control-plane tour: request flow, kubelet, leasesA guided walk through the machinery — what happens between "kubectl apply" and a running container, how the kubelet turns a PodSpec into containers via the CRI, and how leader election keeps one controller-manager in charge. Reading + live pokes on this cluster; no single check to pass.
Lesson: CRDs: teach the API server a new nounA team shipped manifests for a BackupSchedule resource — and the apply fails with "no matches for kind". Nothing is broken: the cluster just doesn't know that noun yet. Install the CustomResourceDefinition, watch the API grow a new endpoint, and see how CRD + controller = operator.
Lesson: etcd backup & restore: the runbook you rehearse before you need itActually do it: snapshot etcd, destroy the object, restore it back. The cert flags everyone fumbles, the restore's new-cluster semantics, and why Reddit's Pi-Day recovery took five hours. The last lesson of "etcd is the cluster" — knowing how to put it back, rehearsed rather than read.
Lesson: Admission: the API server edits your YAML before storing itApply a bare pod and read back what the cluster stored: resources you never wrote, a ServiceAccount you never named, volumes you never mounted. That's the admission chain rewriting objects in flight. Drive it yourself with a LimitRange — defaults in (mutation), oversized pods out (validation).
Lesson: Watch: how every controller hears the clusterReading with live experiments — the list+watch mechanism underneath every controller, kubectl -w, and the scheduler alike: resourceVersion as a cursor, watch events as a change feed, informers as the cache that makes 10,000 controllers cheap, and API Priority & Fairness as the API server's self-defense when they aren't.
Lesson: Build an operator: the verb for your nounReading — the capstone for internals. You created the CRD (7.5); now walk the controller that would bring it to life: informer, workqueue, reconcile(key), status writes, ownerReferences. Annotated minimal code, the crash-safety argument, and the honest ladder from client-go to kubebuilder — including when not to write an operator at all.
Lesson: kubeadm: a cluster from three commandsThe full kubeadm init → join runbook: what each phase does, the pod-network-cidr decision you can't take back, join tokens and their expiry, and the CNI step everyone forgets. Then run the join half for real — init tears node-02 out of the cluster, and you mint a fresh join command and bring it back Ready, the exact TLS-bootstrap flow a new node takes.
Lesson: HA control plane: three of everythingWhat it takes for the control plane to survive losing a node: stacked vs external etcd, the --control-plane-endpoint decision, joining more control-plane nodes, who load-balances the apiserver, and how the singleton controllers pick a leader. Then run the leader-election half for real — kill the scheduler's Lease holder and prove a new identity acquires it.
Lesson: Certificates: the outage scheduled a year in advancekubeadm clusters mint one-year certificates, and expired certs are a whole-cluster lockout that arrives on schedule. Run the real drill on a real control plane: check-expiration, renew all, restart the static pods, re-copy admin.conf — and prove the apiserver is actually serving the new certificate, not just holding it on disk.
Module 8: Observability & SRERunning it for real: metrics-server, events as forensics, logs, node NotReady triage, disk and memory pressure, eviction, and the debugging playbooks that turn a 3 a.m. page into a five-minute fix.
Lesson: Events forensics: reconstruct the crimeA pod is gone and nobody knows why. No logs survive a deleted pod — but the cluster's events remember. Learn to read events as a timeline, the tool that turns "it just disappeared" into a precise cause.
Lesson: Incident replay — the node that OOMed itself (Blue Matador)Replay of Blue Matador's cited postmortem: pods with no memory limits grew until the NODE ran out, and the kernel OOM-killer started shooting processes — including system daemons — at random. Learn node-level OOM vs pod OOM, and cap the workload before the kernel does it for you.
Lesson: Deploy blocked: the quota nobody mentionedA scale-up half-worked: 2 of 5 pods created, then FailedCreate, and the Deployment is stuck. A ResourceQuota is out of room — and the error is on the ReplicaSet, not the pods. Learn to read quota, and the layer where this class of failure hides.
Lesson: The morning after NotReadyA worker went NotReady overnight — kubelet certificate expired, on-call fixed it at 3 a.m. and went back to bed. The node is healthy now, but half your capacity is still gone and pods are Pending. Learn what NotReady actually does to a node — taints, evictions, cordons — and how to clean up after it.
Lesson: Pattern drill: evicted — the disk you forgot to budgetPods keep dying with a status you haven't seen before: Evicted. No crash, no OOM, exit code nowhere — the kubelet itself killed them for eating too much disk. Ephemeral storage is the resource nobody puts in their manifests, and eviction is how you find out.
Lesson: Incident file — the OS under the cluster (Datadog, 2023)Guided study of Datadog's cited 2023 outage: a routine systemd security update auto-applied across the fleet restarted systemd-networkd, which silently deleted the network routes Cilium had installed — and tens of thousands of nodes across five regions dropped off the network in the same hour. The layer below Kubernetes can take down everything above it.
Lesson: The upgrade runbook: version skew, kubeadm, and not becoming a war storyReading — how production Kubernetes clusters are upgraded without downtime: the version-skew contract that makes rolling upgrades possible, the kubeadm sequence (plan, apply, drain, uncordon, repeat), what breaks anyway, and the pre-flight checklist distilled from the incidents in this course.
Lesson: Node maintenance: drain like you mean itA worker node needs a kernel patch. It runs a DaemonSet pod and a batch pod with emptyDir scratch data — the two things kubectl drain refuses to touch without explicit flags. Run the full maintenance cycle: cordon, drain past both refusals, "patch", uncordon, and leave a paper trail.
Lesson: SLOs: the burn-rate alert that never firedAn SLO burn-rate alert has been silently broken for months: its PrometheusRule references a metric that doesn't exist, and the "error ratio" counts 404s as server errors. Fix both bugs and prove the burn rate evaluates against live traffic.
Lesson: OTel pipeline: traces into the voidAn OpenTelemetry Collector receives traces from every service and exports them to Jaeger — except Jaeger has been empty for a week. The collector's exporter points at a service name that doesn't exist, and it retries into the void. Read the pipeline config, fix the endpoint, watch traces land.
Lesson: Drill — the log pipeline that ships nothingSynthetic drill of a failure pattern reported across many production clusters: the app logs, the shipper runs, every pod is Ready — and the log backend has been empty for a week. A tail input matching a path that never existed drops every record silently. Find the gap between "the agent is healthy" and "the records arrived".
Module 9: War Stories — capstone incidentsMulti-concept cascade failures reproduced from real, cited production postmortems — incidents that cross DNS, scheduling, scaling and control-plane boundaries at once. Everything you've learned, under fire. Single-concept incident labs live in their concept modules; this is the boss level.
Lesson: Incident file — the cascade (Monzo's bank-stopping outage)Guided study of Monzo's cited outage: a routine change touched etcd, which confused a service mesh, which returned empty endpoints, which crashed clients on a null pointer — and a bank's payments stopped. Trace how five familiar mechanisms compose into one national-scale failure.
Lesson: Incident file — locked out of the control plane (OpenAI, 2024)Guided study of OpenAI's cited December 2024 outage: a fleet-wide telemetry rollout overwhelmed every Kubernetes API server at once, DNS caching hid the damage until the rollout was everywhere, and the overloaded control plane locked engineers out of the fix. Control plane vs data plane, as a war story.
Lesson: Incident file — the Pi-Day outage (Reddit, 2023)Guided study of Reddit's cited Pi-Day outage: a routine Kubernetes upgrade on their oldest cluster renamed one node label — and the CNI's route reflectors, selecting nodes by the old name, went empty. Pod networking collapsed cluster-wide. A war story about deprecations, snowflake clusters, and why restores get tested before they're needed.
Lesson: Incident file — killing the dashboard on Black Friday (Algolia)Guided study of Algolia's cited war story: a Jobs-based workload meets the biggest traffic day of the year, work piles up faster than it drains, and the accumulated load takes the platform down at peak. The overload playbook — backpressure, cleanup, and capacity math — as a story with a date on it.
Lesson: Incident file — the five-system cascade (Target, 2019)Guided study of Target's cited cascading failure: an OpenStack network upgrade made Kafka flap, logging sidecars stampeded, Docker daemons buckled, Kubernetes rescheduling poisoned Consul's gossip mesh with 41,000 phantom nodes, and Vault sealed itself. A war story about cascades that refuse to stay inside one system's boundary.
Lesson: Incident file — deleting every cluster, twice (Spotify, 2019)Guided study of Spotify's cited KubeCon keynote: during the GKE migration, production Kubernetes clusters were accidentally deleted — twice — with little to no user impact. The war story where the interesting question is why nothing happened, and what that says about where cluster state should live.
Lesson: Final boss: three faults, no hintsA checkout stack with three independent faults stacked on top of each other — a selector mismatch, a bad probe, and an impossible resource request. No hints, no fault list. Diagnose each from first principles and ship the fix. This is the exam.
Module 10: Platform EngineeringBuilding and operating Internal Developer Platforms on Kubernetes: GitOps with ArgoCD and Flux, multi-tenancy with Capsule, cluster lifecycle with Cluster API, and self-service infrastructure with Crossplane. The layer where one team's Kubernetes becomes every team's platform.
Lesson: Argo CD: the app that refuses to syncArgo CD is installed and an Application points at the guestbook example — but the repo path in its spec doesn't exist, so it can't even compare desired vs live. Read the Application status like an operator, fix the source, and watch auto-sync converge it to Synced/Healthy.
Lesson: App of apps: one bad child in the fleetThree Applications deploy the shop's frontend stack in sync-wave order — and the fleet is stuck: one child pins its targetRevision to a branch that doesn't exist in the repo. Triage a multi-app rollout, fix the bad revision, and get every Application to Synced/Healthy.
Lesson: Flux: the Kustomization that can't find its sourceFlux reconciles podinfo from git — except the GitRepository pins a branch that doesn't exist, so the artifact never materializes and the Kustomization downstream starves. Read Flux's condition chain source → kustomization, fix the ref, watch it converge.
Lesson: Capsule: the tenant that hit its wallsMulti-tenancy without a cluster per team: Capsule carves one cluster into tenants. team-alpha was sized for a single namespace back when it was a single service — now every attempt to add one bounces off the tenant quota. Resize the tenant, prove the walls still hold between teams.
Lesson: Cluster API: clusters as custom resourcesReading — the endgame of the operator pattern: a management cluster whose CRDs describe entire Kubernetes clusters, with providers reconciling machines the way a Deployment reconciles pods. Architecture, the core objects, a real MachineDeployment, and when CAPI beats kubeadm-by-hand.
Lesson: Crossplane: the composition missing its providerA platform team publishes XDatabase as self-service API — claim one and a Composition assembles the real resources. Except every XDatabase sits Synced=False: the Composition emits a resource kind no installed provider understands. Read the composition error, install the missing provider, watch the claim go Ready.
Lesson: Incident replay — a couple of characters pruned the world (Skyscanner)Replay of Skyscanner's cited 2021 global outage: a rarely-touched config file lost its template delimiters, rendered to nothing, and the GitOps reconciler did exactly as instructed — it pruned every Service it managed, worldwide. Restore the config, then make the reconciler fail closed so an empty render can never be mistaken for "delete everything".
Lesson: Incident file — how we failed to integrate Istio (Exponea)Read-through of Exponea's cited write-up: a service-mesh rollout abandoned after the sidecar broke Jobs, StatefulSets, gRPC load balancing and graceful shutdown one after another. No tasks; this is the platform-engineering judgement call — what a mesh actually costs, which of your workloads it changes, and how to decide whether to adopt one at all.
Module 11: Node & Control Plane — failures a container can't stageThe faults that live below kubectl: a kubelet or containerd that won't start, a crash-looped kube-apiserver, an etcd NOSPACE alarm to compact and defrag, static pods, pending node CSRs, a cgroup-driver mismatch, clock skew, and the kernel sysctls pod networking needs. Every drill needs a real machine with systemd, a kernel, and root — so these run on iximiuz Labs only, never local kind.
Lesson: kubelet down: a node goes NotReady from the host upA worker's kubelet won't start — the node has gone NotReady and its pods are stranded. This is the failure M8's node-notready triage points at but can't reach: someone left an invalid flag in the kubelet's extra-args file, not anything the API can see. Diagnose it with journalctl and systemctl the way you would at 3am, clear the bad config, and bring the node back Ready.
Lesson: containerd down: the CRI is gone and nothing will scheduleThe container runtime on a worker won't start, so the kubelet has no CRI to drive — pods stay Pending or dead and crictl can't connect. You'll diagnose from the runtime side (crictl, systemctl, journalctl), clear the broken containerd config, and watch the node accept workloads again.
Lesson: you broke the control plane: kube-apiserver in a crash loopA bad flag in the kube-apiserver static-pod manifest sends it into a crash loop and the API server is gone — kubectl times out cluster-wide. This is the exam's "you broke the control plane, fix it" task: with no API to query, you drop to crictl to read the container's logs, find the flag it's dying on, correct the manifest, and watch the kubelet bring the API back.
Lesson: etcd NOSPACE: the alarm that makes your cluster read-onlyetcd has tripped its backend quota — the "mvcc: database space exceeded" alarm is armed and the whole cluster has gone read-only: no writes, no new pods, nothing. This is the etcd failure that isn't backup/restore (that's 7.6). You'll read the alarm, compact old revisions, defragment the backend to reclaim disk, right-size the quota, and disarm — the exact NOSPACE runbook.
Lesson: static pods: author one by hand, find its mirrorThe classic exam task and the mechanism the whole control plane is built on: drop a pod manifest into /etc/kubernetes/manifests on a node and the kubelet runs it directly — no scheduler, no API needed to start it — then publishes a read-only "mirror" object so the pod shows up in kubectl. You'll write one, watch the kubelet pick it up, and confirm the mirror.
Lesson: pending CSRs: approve a node into the clusterA kubelet's serving-certificate request is sitting Pending — and unlike client certs, kubelet-serving CSRs are never auto-approved, so nothing issues it until you do. This is the manual-approval step operators hit on real clusters, tied to the same TLS-bootstrap machinery behind node joins and cert rotation. You'll read the pending CSR, verify what it's asking for, approve it, and confirm the certificate was issued.
Lesson: cgroup driver mismatch: kubelet and containerd disagreeA node's kubelet and its container runtime are set to different cgroup drivers — the single most common kubeadm install gotcha. It's a latent fault: the node can look Ready while resource accounting is quietly split across two schemes, which is unsupported and bites under memory pressure. You'll detect the disagreement by reading both configs and align them to systemd, the correct driver on a systemd host.
Lesson: clock skew: when a node time-travels out of the clusterA worker's clock has drifted far from real time, and suddenly its kubelet can't talk to the API server — TLS handshakes fail with "certificate has expired or is not yet valid" even though every cert is fine. Time is a security input, and a skewed node is an untrusted node. You'll spot the skew, restore time synchronisation, and watch the node rejoin.
Lesson: pod networking dead: the kernel knobs kubeadm needsPods on a node can't route and Service traffic isn't being NAT'd — because the kernel prerequisites every kubeadm node depends on are off: the br_netfilter module isn't loaded and ip_forward / bridge-nf-call-iptables are 0. You'll restore the module and the sysctls at runtime AND persist them so they survive a reboot — the install gotcha that bites people on the second boot.

About the author

Madhan Kumaravelu

Find this author online

How to Author Courses on iximiuz Labs

Write content locally with your favorite editor or IDE, then sync it to iximiuz Labs using labctl, the platform's command-line tool.

Install labctl CLI

curl -sf https://labs.iximiuz.com/cli/install.sh | sh

This will download and install the latest version of the labctl CLI. You only need to do this once per workstation.

Authorize labctl

labctl auth login

This will open a browser window asking you to authorize labctl to access your account. Sign in after installing labctl and again whenever your session expires.

Pull course content

labctl content pull course kubelings-dbd840c8

This will create a local copy of the course content in a directory named kubelings-dbd840c8. You only need to do this once per course.

Stream your changes

labctl content push -fw course kubelings-dbd840c8

Run this command in a separate terminal to continuously upload your changes to the server while editing the course in your favorite text editor or IDE.

You can also use labctl to create, list, and delete your content. Learn more about the available commands: labctl content --help