Challenge, Hard,  on  Kubernetes, Networking

Migrate the Kubernetes Cluster DNS Domain from cluster.local to iximiuz.cluster

Omkar Shelke
by  Omkar Shelke

Scenario

You are a Kubernetes administrator setting up a cluster for the iximiuz platform. The default Kubernetes DNS domain cluster.local must be changed to iximiuz.cluster to match the platform's internal service discovery naming convention.

All services in the cluster should be reachable using the new domain. For example, the nginx service in the domain namespace must resolve at:

http://nginx.domain.svc.iximiuz.cluster

The nginx deployment with 4 replicas is already running in the domain namespace and is ready to serve traffic. Your job is to reconfigure the cluster DNS domain so that service discovery works under the new iximiuz.cluster zone.

Task

Three components control the cluster DNS domain and all must be updated together. Missing any one of them will either break DNS resolution immediately or cause failures when new nodes join the cluster in the future.

CoreDNS
kubelet
kubeadm-config

CoreDNS is the cluster DNS server. Its configuration lives in the coredns ConfigMap in kube-system.

Every cluster.local reference inside the Corefile must be changed to iximiuz.cluster: the zone argument on the kubernetes plugin line, and the two disable lines inside the cache block. After editing, restart CoreDNS to apply the change.

Hint 1 - Update the CoreDNS ConfigMap

CoreDNS reads its configuration from the coredns ConfigMap in kube-system. Edit it and replace every occurrence of cluster.local with iximiuz.cluster:

kubectl edit configmap coredns -n kube-system

There are three separate lines inside the Corefile that mention cluster.local, not just one. All three need to change, or the ConfigMap check will keep failing even after you fix the others:

  1. The zone argument on the kubernetes plugin line: 
    kubernetes cluster.local in-addr.arpa ip6.arpa {
  2. Two lines inside the cache block: 
    cache 30 {
    disable success cluster.local
    disable denial cluster.local
}

After saving, restart CoreDNS to pick up the new config:

kubectl rollout restart deployment/coredns -n kube-system
kubectl rollout status deployment/coredns -n kube-system

Documentation

kubelet is the node agent on every node. kubelet's clusterDomain setting tells it what DNS suffix to append when resolving service names. This must be updated on both cplane-01 and node-01. After editing, restart kubelet on each node.

After updating kubelet on both nodes, restart the nginx Deployment in the domain namespace. Existing pods were created with cluster.local in their /etc/resolv.conf and must be recreated to get iximiuz.cluster injected by kubelet.

Once the nginx Deployment has been successfully restarted, verify that DNS resolution works end-to-end using:

kubectl run curl-test \
  --rm -it \
  --image=curlimages/curl:8.7.1 \
  --restart=Never \
  -n default -- \
  curl http://nginx.domain.svc.iximiuz.cluster
Hint 2 - Update kubelet clusterDomain on Both Nodes

kubelet reads its configuration from /var/lib/kubelet/config.yaml. Edit this file on both cplane-01 and node-01 and change clusterDomain:

sudo vi /var/lib/kubelet/config.yaml

Find the line:

clusterDomain: cluster.local

Change it to:

clusterDomain: iximiuz.cluster

Then restart kubelet on each node:

sudo systemctl restart kubelet
sudo systemctl status kubelet

Repeat on both nodes. After both kubelet restarts, verify nodes are still Ready:

kubectl get nodes

Documentation

Hint 3 - Restart nginx and CoreDNS

After updating kubelet on both nodes, restart the nginx Deployment so new pods are created with iximiuz.cluster injected into their /etc/resolv.conf. Also restart CoreDNS to make sure it has picked up the updated Corefile:

kubectl rollout restart deployment nginx -n domain
kubectl rollout restart deployment coredns -n kube-system
kubectl rollout status deployment nginx -n domain

Verify the new pods have the correct DNS search domain injected:

POD=$(kubectl get pod -n domain -l app=nginx -o jsonpath='{.items[0].metadata.name}')
kubectl exec "$POD" -n domain -- cat /etc/resolv.conf

You should see iximiuz.cluster in the search line:

nameserver 10.96.0.10
search domain.svc.iximiuz.cluster svc.iximiuz.cluster iximiuz.cluster
options ndots:5

Documentation

Hint 4 - Test DNS Resolution

After updating CoreDNS and kubelet on both nodes, run a one-off curl pod to verify the new domain resolves correctly:

kubectl run curl-test \
  --rm -it \
  --image=curlimages/curl:8.7.1 \
  --restart=Never \
  -n default -- \
  curl http://nginx.domain.svc.iximiuz.cluster

You should see the nginx response containing:

🙂 CoreDNS is configured to resolve services in the iximiuz.cluster zone.

If curl fails, check CoreDNS logs for DNS resolution errors:

kubectl logs -l k8s-app=kube-dns -n kube-system

Documentation

kubeadm-config is the ConfigMap that kubeadm reads to configure kubelet on a node when it joins the cluster via kubeadm join.

If dnsDomain is not updated here, any future node will join with cluster.local instead of iximiuz.cluster, causing DNS failures for all pods on that node. Update the cluster configuration to reflect the new DNS domain and ensure consistent DNS settings across all nodes.

Hint 5 - Update kubeadm-config for Future Node Joins

The kubeadm-config ConfigMap records the cluster settings used by kubeadm when new nodes join. Update dnsDomain so any future node gets the correct kubelet configuration automatically:

kubectl edit configmap kubeadm-config -n kube-system

Under ClusterConfiguration, find networking and update the dnsDomain field to match the new cluster domain.

Documentation

Test Cases

How to Author Challenges on iximiuz Labs

Instead of providing a subpar online editing experience, iximiuz Labs offers a helper CLI tool called labctl, allowing you to use your favorite text editor (or a full-featured IDE) to write content from the comfort of your local machine.

Install labctl CLI

curl -sf https://labs.iximiuz.com/cli/install.sh | sh

This will download and install the latest version of the labctl CLI. You only need to do this once per workstation.

Authorize labctl

labctl auth login

This will open a browser window asking you to authorize labctl to access your account. You need to do it after a fresh install of labctl and repeat it whenever the auth session expires.

Pull challenge content

labctl content pull challenge migrate-cluster-dns-domain-cluster-local-to-iximiuz-cluster-467d9f1c

This will create a local copy of the challenge content in a directory named migrate-cluster-dns-domain-cluster-local-to-iximiuz-cluster-467d9f1c. You only need to do this once per challenge.

Stream your changes

labctl content push -fw challenge migrate-cluster-dns-domain-cluster-local-to-iximiuz-cluster-467d9f1c

Run this command in a separate terminal to continuously upload your changes to the server while editing the challenge in your favorite text editor or IDE.

You can also use labctl to create, list, and delete your content. Learn more about the available commands: labctl content --help

Start challenge