Migrate the Kubernetes Cluster DNS Domain from cluster.local to iximiuz.cluster

Scenario

You are a Kubernetes administrator setting up a cluster for the iximiuz platform. The default Kubernetes DNS domain cluster.local must be changed to iximiuz.cluster to match the platform's internal service discovery naming convention.

All services in the cluster should be reachable using the new domain. For example, the nginx service in the domain namespace must resolve at:

http://nginx.domain.svc.iximiuz.cluster

The nginx deployment with 4 replicas is already running in the domain namespace and is ready to serve traffic. Your job is to reconfigure the cluster DNS domain so that service discovery works under the new iximiuz.cluster zone.

Task

Three components control the cluster DNS domain and all must be updated together. Missing any one of them will either break DNS resolution immediately or cause failures when new nodes join the cluster in the future.

kubectl edit configmap coredns -n kube-system

kubectl rollout restart deployment/coredns -n kube-system
kubectl rollout status deployment/coredns -n kube-system

kubectl run curl-test \
  --rm -it \
  --image=curlimages/curl:8.7.1 \
  --restart=Never \
  -n default -- \
  curl http://nginx.domain.svc.iximiuz.cluster

sudo vi /var/lib/kubelet/config.yaml

clusterDomain: cluster.local

clusterDomain: iximiuz.cluster

sudo systemctl restart kubelet
sudo systemctl status kubelet

kubectl get nodes

kubectl rollout restart deployment nginx -n domain
kubectl rollout restart deployment coredns -n kube-system
kubectl rollout status deployment nginx -n domain

POD=$(kubectl get pod -n domain -l app=nginx -o jsonpath='{.items[0].metadata.name}')
kubectl exec "$POD" -n domain -- cat /etc/resolv.conf

nameserver 10.96.0.10
search domain.svc.iximiuz.cluster svc.iximiuz.cluster iximiuz.cluster
options ndots:5

kubectl run curl-test \
  --rm -it \
  --image=curlimages/curl:8.7.1 \
  --restart=Never \
  -n default -- \
  curl http://nginx.domain.svc.iximiuz.cluster

🙂 CoreDNS is configured to resolve services in the iximiuz.cluster zone.

kubectl logs -l k8s-app=kube-dns -n kube-system

kubectl edit configmap kubeadm-config -n kube-system

Test Cases

Understanding the Challenge

This challenge requires changing the Kubernetes cluster DNS domain from cluster.local to iximiuz.cluster. To understand why this is non-trivial, you first need to understand how DNS works inside a Kubernetes cluster and which components are responsible for it.

How DNS Works in Kubernetes

When you create a Service in Kubernetes, it gets a cluster IP (for example 10.96.45.23). Pods could theoretically talk to each other using these IPs directly, but cluster IPs change every time a Service is deleted and recreated. Hardcoding IPs in application code breaks constantly.

Kubernetes solves this with DNS-based service discovery. Instead of using IPs, pods communicate using predictable DNS names like:

nginx.domain.svc.cluster.local

This name is stable: it never changes even if the Service is recreated with a different IP.

The full DNS name format for a Service is:

<service-name>.<namespace>.svc.<cluster-domain>

So for the nginx Service in the domain namespace with cluster.local as the domain:

nginx.domain.svc.cluster.local  resolves to  10.96.45.23

After changing the domain to iximiuz.cluster:

nginx.domain.svc.iximiuz.cluster  resolves to  10.96.45.23

What Is CoreDNS and How Does It Work

CoreDNS is the DNS server that runs inside the Kubernetes cluster. It is a Deployment in the kube-system namespace, usually with 2 replicas for high availability. CoreDNS watches the Kubernetes API server for Services and Endpoints and automatically builds DNS records for them.

When a pod wants to resolve nginx.domain.svc.cluster.local, it sends a DNS query to CoreDNS. CoreDNS looks up the Service in the cluster and responds with the ClusterIP.

CoreDNS reads its configuration from the coredns ConfigMap in kube-system. This file is called the Corefile:

.:53 {
    errors
    health {
       lameduck 5s
    }
    ready
    kubernetes cluster.local in-addr.arpa ip6.arpa {
       pods insecure
       fallthrough in-addr.arpa ip6.arpa
       ttl 30
    }
    prometheus :9153
    forward . /etc/resolv.conf {
       max_concurrent 1000
    }
    cache 30 {
       disable success cluster.local
       disable denial cluster.local
    }
    loop
    reload
    loadbalance
}

Breaking down the Corefile line by line:

.:53: CoreDNS listens on port 53 (the standard DNS port) for all queries. The . means root, matching every DNS query regardless of domain.

errors: log DNS errors to stdout so they appear in kubectl logs.

health: expose a health check endpoint at :8080/health. The lameduck 5s means when CoreDNS is shutting down, it continues serving requests for 5 seconds to drain in-flight queries gracefully before terminating.

ready: expose a readiness endpoint at :8181/ready. Kubernetes uses this for the readiness probe on the CoreDNS pod; it only receives traffic once this returns OK.

kubernetes cluster.local in-addr.arpa ip6.arpa: this is the most important block. This tells CoreDNS to handle DNS queries for the cluster.local domain and reverse DNS lookups for IPv4 (in-addr.arpa) and IPv6 (ip6.arpa). It talks directly to the Kubernetes API server to resolve Service and Pod names. pods insecure creates DNS A records for individual pod IPs. fallthrough means if a reverse DNS query is not found here, pass it to the next plugin instead of returning NXDOMAIN.

prometheus :9153: expose DNS query metrics at port 9153 in Prometheus format for monitoring.

forward . /etc/resolv.conf: for any DNS query CoreDNS cannot answer internally (e.g. google.com), forward it upstream to the DNS servers listed in /etc/resolv.conf on the CoreDNS pod itself (inherited from the node's network config). max_concurrent 1000 limits concurrent upstream requests.

cache 30: cache DNS responses for up to 30 seconds to reduce load on the API server. disable success cluster.local and disable denial cluster.local disable caching specifically for cluster-internal queries, ensuring that when a new Service is created or a pod is scaled, DNS changes are reflected immediately without waiting for a stale cache entry to expire.

loop: detect and abort DNS forwarding loops to prevent infinite query cycles.

reload: automatically detect and reload the Corefile when the ConfigMap changes, without needing to restart the CoreDNS pods.

loadbalance: round-robin DNS load balancing across multiple pod IPs when a Service has more than one endpoint.

How DNS Is Injected into Every Pod

When kubelet starts a pod, it automatically configures the pod's DNS settings by writing /etc/resolv.conf inside the container. You never configure this manually; kubelet does it for every pod based on its own clusterDomain setting.

Inside every pod, /etc/resolv.conf looks like:

nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

• nameserver 10.96.0.10: the ClusterIP of the kube-dns Service in kube-system, which points to CoreDNS. Every DNS query from the pod goes here first.
• search default.svc.cluster.local svc.cluster.local cluster.local: search domains. If a pod queries just nginx, the OS automatically tries nginx.default.svc.cluster.local, then nginx.svc.cluster.local, then nginx.cluster.local before giving up. This is why short names resolve inside the same namespace without typing the full FQDN.
• options ndots:5: if a hostname has fewer than 5 dots, treat it as relative and apply the search domains above before trying it as an absolute name.

kubelet knows what domain to put in the search list because of the clusterDomain field in /var/lib/kubelet/config.yaml. If clusterDomain is cluster.local, the search list uses cluster.local. If you change it to iximiuz.cluster and restart kubelet, all new pods created after the restart will have iximiuz.cluster in their /etc/resolv.conf search list.

Why All Three Components Must Be Updated

This is the key insight of the challenge. Three separate components control the DNS domain and all must be updated together.

CoreDNS handles the resolution side: it must know to serve DNS records under iximiuz.cluster. If only CoreDNS is updated, the pods' /etc/resolv.conf still has cluster.local in the search list, so short name resolution breaks. Explicit FQDNs with iximiuz.cluster will work but short names will not.

kubelet on every node handles the pod injection side: it writes the search domain into every new pod's /etc/resolv.conf. If only kubelet is updated, pods will try to resolve nginx.domain.svc.iximiuz.cluster but CoreDNS still only answers queries for cluster.local, so DNS fails entirely.

kubeadm-config ConfigMap handles the future node join side: when a new worker node joins the cluster via kubeadm join, kubeadm reads this ConfigMap to generate the kubelet configuration for that node, including clusterDomain. If dnsDomain is not updated here, every future node that joins will be silently misconfigured with cluster.local, causing DNS failures for all pods scheduled on that node.

All three must match. If any one disagrees, DNS breaks, either immediately or the next time a new node joins the cluster.

Step 1: Inspect the Current CoreDNS Configuration

kubectl get configmap coredns -n kube-system -o yaml

Note every place cluster.local appears in the Corefile, there are three occurrences:

1. In the kubernetes plugin line
2. In the cache block, disable success cluster.local
3. In the cache block, disable denial cluster.local

All three must be changed. Missing the cache entries means CoreDNS may still serve stale cluster.local responses after the change.

Step 2: Update the CoreDNS ConfigMap

Edit the ConfigMap directly:

kubectl edit configmap coredns -n kube-system

Change every occurrence of cluster.local to iximiuz.cluster:

# before
kubernetes cluster.local in-addr.arpa ip6.arpa {
   pods insecure
   fallthrough in-addr.arpa ip6.arpa
   ttl 30
}
...
...
cache 30 {
   disable success cluster.local
   disable denial cluster.local
}

# after
kubernetes iximiuz.cluster in-addr.arpa ip6.arpa {
   pods insecure
   fallthrough in-addr.arpa ip6.arpa
   ttl 30
}
...
...
cache 30 {
   disable success iximiuz.cluster
   disable denial iximiuz.cluster
}

The reload plugin detects the ConfigMap change and reloads automatically. Force a restart to apply immediately:

kubectl rollout restart deployment/coredns -n kube-system
kubectl rollout status deployment/coredns -n kube-system

Verify all three were updated using jq:

kubectl get configmap coredns -n kube-system -o json | \
  jq '.data.Corefile | contains("kubernetes iximiuz.cluster")'

kubectl get configmap coredns -n kube-system -o json | \
  jq '.data.Corefile | contains("disable success iximiuz.cluster")'

kubectl get configmap coredns -n kube-system -o json | \
  jq '.data.Corefile | contains("disable denial iximiuz.cluster")'

true

All three should return true.

Step 3: Update kubelet clusterDomain on cplane-01

kubelet's cluster domain is configured in /var/lib/kubelet/config.yaml. Edit it:

sudo vi /var/lib/kubelet/config.yaml

Find and update clusterDomain:

# before
cgroupDriver: systemd                          
clusterDNS:                                    
- 10.96.0.10                                   
clusterDomain: cluster.local

# after
cgroupDriver: systemd                          
clusterDNS:                                    
- 10.96.0.10                                   
clusterDomain: iximiuz.cluster

Verify the change using yq, which reads YAML natively:

sudo yq '.clusterDomain' /var/lib/kubelet/config.yaml

iximiuz.cluster

Restart kubelet to apply:

sudo systemctl restart kubelet
sudo systemctl status kubelet

Step 4: Update kubelet clusterDomain on node-01

SSH to node-01 and repeat the same change:

sudo vi /var/lib/kubelet/config.yaml

cgroupDriver: systemd                          
clusterDNS:                                    
- 10.96.0.10  
clusterDomain: iximiuz.cluster

Verify with yq:

sudo yq '.clusterDomain' /var/lib/kubelet/config.yaml

iximiuz.cluster

Restart kubelet and verify it is running:

sudo systemctl restart kubelet

Step 5: Restart nginx Deployment to Inject New DNS Search Domain

The nginx pods in the domain namespace were created during init setup, before kubelet was updated to iximiuz.cluster. Their /etc/resolv.conf still has cluster.local as the search domain. kubelet does not rewrite /etc/resolv.conf for existing pods, only new pods get the updated search domain injected.

Restart the nginx Deployment so kubelet creates fresh pods with iximiuz.cluster:

kubectl rollout restart deployment nginx -n domain
kubectl rollout status deployment nginx -n domain

Verify the new pods have the correct search domain injected:

POD=$(kubectl get pod -n domain -l app=nginx -o jsonpath='{.items[0].metadata.name}')
kubectl exec "$POD" -n domain -- cat /etc/resolv.conf

search domain.svc.iximiuz.cluster svc.iximiuz.cluster iximiuz.cluster
nameserver 10.96.0.10
options ndots:5

iximiuz.cluster is now in the search list, so the pod will resolve service names via the new domain correctly.

Step 6: Update the kubeadm-config ConfigMap

Update the dnsDomain field so any future node that joins the cluster is configured with the correct domain:

kubectl edit configmap kubeadm-config -n kube-system

Find the networking section under ClusterConfiguration and update dnsDomain:

networking:
  dnsDomain: iximiuz.cluster   # was cluster.local
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.96.0.0/12

Verify using jq:

kubectl get configmap kubeadm-config -n kube-system -o json | \
  jq -e '.data.ClusterConfiguration | contains("dnsDomain: iximiuz.cluster")'

true

Step 7: Verify All Nodes Are Ready

Confirm both nodes are still healthy after the kubelet restarts:

kubectl get nodes

NAME        STATUS   ROLES           AGE     VERSION
cplane-01   Ready    control-plane   7m19s   v1.36.0
node-01     Ready    <none>          7m7s    v1.36.0

Step 8: Verify CoreDNS Is Running with the New Config

kubectl get deployment coredns -n kube-system -o json | \
  jq '{ready: .status.readyReplicas, desired: .status.replicas}'

{
  "ready": 2,
  "desired": 2
}

Step 9: Test End-to-End DNS Resolution

Run a one-off curl pod and query the nginx Service using the new domain:

POD_NAME="dns-verify-$(date +%s)"
kubectl run "$POD_NAME" \
  --rm -it \
  --image=curlimages/curl:8.7.1 \
  --restart=Never \
  -n default -- \
  curl http://nginx.domain.svc.iximiuz.cluster

🙂 CoreDNS is configured to resolve services in the iximiuz.cluster zone.

DNS resolution is working under the iximiuz.cluster domain.