CoreDNS and the internal DNS

You have spent the whole course writing http://web and it just works. Behind that convenience there is a DNS server running inside the cluster itself (CoreDNS), an /etc/resolv.conf file that the kubelet injects into every Pod and a couple of details that cause inexplicable latencies when you do not know them.

Work from the dev-machine tab. Start by saying hello to the server:

kubectl get deployment coredns -n kube-system

kubectl get service kube-dns -n kube-system

kubectl get configmap coredns -n kube-system -o yaml

Three pieces you already know how to read: CoreDNS is a perfectly ordinary Deployment (with its replicas, its rollout and its rollback), exposed by a Service historically called kube-dns, and configured by a ConfigMap with a file called Corefile. That Corefile is a chain of plugins; the one doing the magic is called kubernetes and it is what translates Service and Pod objects into DNS records. And notice the forward plugin: it is the door to the outside DNS, the one that lets a Pod also resolve example.com.

Step 1: The scenario

Bring up a Service in tienda (the namespace the context already leaves you in) and a client to ask questions from (cliente means "client"). By now you can do it with your eyes closed:

kubectl create deployment web --image=ghcr.io/iximiuz/labs/nginx:alpine --port=80
kubectl expose deployment web --port=80 --target-port=80
kubectl run cliente --image=ghcr.io/iximiuz/labs/nginx:alpine --command -- sleep 100000

The playground also brings a namespace otro ("other") with another Service that is also called web. That deliberate collision is the heart of the lesson.

Step 2: The anatomy of a name

Every Service has a full name with this shape:

<service>.<namespace>.svc.<cluster-domain>

That is, web.tienda.svc.cluster.local. The four pieces, from right to left: the cluster domain (configurable, almost always cluster.local), svc (which tells it apart from Pod records), the namespace and the name of the Service.

Ask for it from the client, with the full name and with the short name:

kubectl exec cliente -- nslookup web.tienda.svc.cluster.local

kubectl exec cliente -- nslookup web

Both return the same IP. Why does the short one work? Because of the file the kubelet put inside it:

kubectl exec cliente -- cat /etc/resolv.conf

Three lines worth understanding:

• nameserver: the ClusterIP of the kube-dns Service. That is where the Pod asks.
• search: the list of suffixes the resolver tries when you give it an incomplete name: tienda.svc.cluster.local, svc.cluster.local, cluster.local. That is why web ends up finding web.tienda.svc.cluster.local and not the one in the otro namespace: the first suffix on the list is the Pod's own namespace.
• options ndots:5: and here is the classic of DNS in Kubernetes.

What does ndots:5 mean? That if the name you ask for has fewer than 5 dots, the resolver considers it "incomplete" and tries all the search suffixes first before trying it as is. With web that is what we want. But look at what happens with an external name:

example.com has 1 dot, that is, fewer than 5. So the Pod asks, in this order:

1. example.com.tienda.svc.cluster.local → does not exist
2. example.com.svc.cluster.local → does not exist
3. example.com.cluster.local → does not exist
4. example.com → at last

Three failed queries before every hit, on every resolution, multiplied by all the requests of your application. It is one of the classic causes of inexplicable latency and of CoreDNS overload in large clusters. The trick to avoid it is as simple as it is ugly: put a dot at the end of the name (example.com.), which declares it absolute and skips the whole search. Try it:

kubectl exec cliente -- nslookup example.com

kubectl exec cliente -- nslookup example.com.

Both answer the same, and that is precisely the trap: the three failed queries are not visible. They happen underneath, inside the container's resolver, and nslookup only shows the one that hit. To see them you have to look from the other side, in the CoreDNS logs or by capturing the Pod's traffic. What changes between the two forms is not the result: it is the work it took to get there.

Step 3: Across namespaces

Now the collision. From the client, which lives in tienda, ask for both:

kubectl exec cliente -- nslookup web

kubectl exec cliente -- nslookup web.otro

kubectl exec cliente -- nslookup web.otro.svc.cluster.local

The short name stays home; the name with the namespace crosses the border. This is the mechanism that makes namespaces true name spaces: two teams can call their services web without stepping on each other, and each one resolves its own by default.

Step 4: ExternalName, the Service that has no Pods

Last piece, and the oddest of the Service family: one that selects nothing and spreads no traffic. Create externo.yaml (externo means "external"):

cat << 'EOF' > externo.yaml
apiVersion: v1
kind: Service
metadata:
  name: externo
spec:
  type: ExternalName
  externalName: example.com
EOF

The YAML, in three questions

Where is the selector?

There is none, and there is no ClusterIP either, nor endpoints, nor kube-proxy in the middle. An ExternalName is not a proxy: it is an entry in the DNS. When someone asks for externo.tienda.svc.cluster.local, CoreDNS answers with a CNAME to example.com and washes its hands of it. The client makes the connection directly against the destination.

So what is it for?

To give a stable name inside the cluster to something that lives outside: the cloud provider's managed database, a third-party API, a legacy system. Your application always talks to db.tienda.svc.cluster.local, and the day that database migrates into the cluster, you change the Service from ExternalName to a normal one with a selector and the application does not notice. It is a free layer of indirection.

Any catch?

Two. Being a CNAME, it does not work well with protocols that are not name-based (for example, a client validating a TLS certificate will see the external name, not the internal one). And since it does not go through kube-proxy, egress NetworkPolicies do not filter it by Service: the traffic goes out to some external IP.

kubectl apply -f externo.yaml

kubectl get service externo

kubectl exec cliente -- nslookup externo.tienda.svc.cluster.local

Notice the CLUSTER-IP column: empty. And the answer of the nslookup: a CNAME.

Summary

• CoreDNS is just another Deployment, with its ConfigMap (the Corefile) and its kube-dns Service. It can be scaled, updated and broken like any other.
• The canonical name is <service>.<namespace>.svc.cluster.local; the short name works thanks to the search list in /etc/resolv.conf.
• ndots:5 makes external names generate extra failed queries. The trailing dot (example.com.) avoids them.
• Namespaces are true name spaces: two Services can have the same name.
• ExternalName is a CNAME with a Service name: the indirection for what lives outside the cluster.