Kubernetes DNS and Service Discovery: CoreDNS, ndots:5 and the Latency Tax (2026)

A pod calls api.partner.com once. The application logs one lookup. On the wire, the resolver sends eight DNS queries, six of which are guaranteed to fail, and every one of them crosses the node’s network stack, a Service virtual IP and a conntrack table before reaching CoreDNS. Multiply that by a few thousand pods, and DNS becomes the busiest and least-monitored dependency in the cluster.

Kubernetes DNS is deceptively simple in the happy path: a Service name resolves to a ClusterIP, and everything works. The cost hides in three defaults, the ndots:5 resolver option, the search-domain list, and the UDP path through kube-proxy’s NAT. Together they produce query amplification, intermittent five-second stalls and upstream throttling that only appears under load.

This article walks through exactly what a pod’s resolver does, how CoreDNS processes each query, where the latency tax comes from, and which fixes (fully qualified names, autopath, NodeLocal DNSCache, tuned dnsConfig) actually pay off. You will leave with a worked query count, a tuned Corefile, and a debugging runbook.

What this covers: pod resolver behaviour and ndots, the CoreDNS plugin chain, the conntrack race behind five-second timeouts, NodeLocal DNSCache, headless Services, SRV records, EndpointSlices, ExternalName, DNS-based versus xDS-based service discovery, CoreDNS autoscaling, a debugging workflow, and failure modes such as negative caching and cloud resolver limits.

Context and Background

Service discovery answers one question: given a stable name, which network endpoints should I talk to right now? Kubernetes answers it in two layers. The control plane maintains Services and EndpointSlices, which map a stable virtual IP and port to the changing set of ready pods. DNS sits on top and maps human-friendly names to those virtual IPs, or, for headless Services, directly to pod IPs.

The authoritative description is the Kubernetes documentation on DNS for Services and Pods. A Service named payments in namespace shop gets an A or AAAA record at payments.shop.svc.cluster.local, where cluster.local is the default cluster domain. Named ports additionally get SRV records of the form _port-name._protocol.payments.shop.svc.cluster.local. Pods can get records of the form 10-244-1-5.shop.pod.cluster.local, although the CoreDNS Kubernetes plugin disables pod records unless you ask for them.

Since Kubernetes 1.11, CoreDNS has been the default cluster DNS server, replacing kube-dns, and it is a CNCF graduated project. It is a plugin-chain DNS server written in Go: each server block in the Corefile lists plugins, and a query flows through them in a fixed compile-time order, not the order written in the file. That detail matters later when we discuss caching and autopath.

The kubelet is what wires pods to it. For each pod the kubelet writes /etc/resolv.conf according to the pod’s dnsPolicy. With the default ClusterFirst, the file points at the cluster DNS Service IP (commonly 10.96.0.10 on kubeadm clusters, though the address is cluster specific), adds three Kubernetes search domains, and sets options ndots:5. A pod with dnsPolicy: Default inherits the node’s resolver configuration instead, and None means “use only what dnsConfig supplies”.

Most teams only notice DNS when it breaks. The symptoms are familiar: sporadic five-second delays on outbound calls, getaddrinfo EAI_AGAIN errors in Node.js, UnknownHostException in Java, CoreDNS pods pegged on CPU, and a cloud provider quietly dropping packets at its resolver. Each has a mechanism, and each mechanism is cheaper to understand than to guess at.

This post sits next to our networking coverage. The datapath that carries these queries is discussed in the CNI comparison of Calico, Cilium, Flannel and Multus, and the identity-aware proxying that can sit on top of DNS is covered in the Cilium service mesh deep dive.

A note on scope and honesty. Specific numbers here come from vendor documentation (CoreDNS cache defaults, the AWS link-local packet quota) or from arithmetic you can reproduce. Where a figure depends on your cluster, such as queries per second or CPU per CoreDNS replica, I label it illustrative and show how to measure your own.

How Kubernetes DNS Works: Resolver, Search Domains and CoreDNS

Direct answer: In Kubernetes DNS, each pod’s resolver sends queries to the cluster DNS Service (CoreDNS). The pod’s resolv.conf lists search domains and ndots:5, so any name with fewer than five dots is first tried with each search suffix appended. CoreDNS answers cluster names from the Kubernetes API and forwards everything else upstream.

Figure 1: The Kubernetes DNS path. The pod resolver appends search domains, the query reaches a Service VIP, kube-proxy or the CNI load-balances it to a CoreDNS pod, and CoreDNS either answers from the API cache or forwards upstream.

Figure 1 shows the whole path. The pod’s libc or musl resolver builds a candidate list of names, sends each as a UDP datagram to the nameserver address, and waits. That nameserver address is a ClusterIP, which means the packet is destination-NATted by kube-proxy (iptables or IPVS) or by an eBPF datapath to one of the CoreDNS pod IPs. CoreDNS then runs the plugin chain and replies along the reverse path.

The pod resolver: search, ndots and the candidate list

A typical resolv.conf inside a pod in namespace shop looks like this:

nameserver 10.96.0.10
search shop.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

The ndots option sets a threshold. If the name being looked up contains at least ndots dots, the resolver tries it as an absolute name first and falls back to the search list only if that fails. If it has fewer dots, the resolver tries the search list first and the bare name last. With ndots:5, a name such as api.partner.com has two dots, so it is treated as relative. A name with a trailing dot, api.partner.com., is always absolute and skips the search list entirely.

Why does Kubernetes choose 5? Because in-cluster names get long. The fully qualified Service name payments.shop.svc.cluster.local has four dots, so it is just under the threshold and developers can also write payments.shop (one dot) or payments (zero dots) and still get the search list. The default is a compromise designed for in-cluster convenience, and it taxes every external lookup.

Search-domain handling is also constrained by the C library. glibc historically caps the number of search domains and total search-line length, and Kubernetes warns when the generated list exceeds those limits; if your node’s own search domains are appended to the three Kubernetes ones, you can hit the cap and silently lose entries. Check the Kubernetes DNS documentation for the exact current limits for your version rather than trusting a number from memory.

A worked query count

Count the packets, not the lookups. Take a pod in namespace shop on a cluster whose nodes add no extra search domains. Each lookup below is performed by getaddrinfo, which by default issues both an A and an AAAA query for each candidate name unless the application or options single-request changes it.

Name the app looks up	Dots	Candidates tried (in order)	Failed candidates	Total queries (A + AAAA)
payments	0	payments.shop.svc.cluster.local hits	0	2
payments.billing	1	.shop.svc... NXDOMAIN, .svc.cluster.local hits	1	4
api.partner.com	2	three search suffixes NXDOMAIN, then bare name	3	8
api.partner.com.	3 (absolute)	bare name only	0	2
a.b.c.d.example.com	5	bare name first (threshold met)	0	2

The external name needs eight queries to get two useful answers, a 4x amplification, and six of the eight are NXDOMAIN responses that exist only because of the search list. If the node’s resolv.conf contributes a cloud-provider domain such as ec2.internal or a corporate suffix, the search list gets longer and the amplification rises to 10 or 12 queries per external lookup.

Illustrative scale: a service making 200 distinct external lookups per second across a fleet, with no application-level caching, sends 1,600 packets per second to CoreDNS where 400 would suffice. Your numbers will differ. The point is that the multiplier is fixed by configuration, not by traffic shape, so it scales linearly with every new microservice that calls an external API.

What CoreDNS does with each query

CoreDNS receives the query on port 53 (UDP and TCP) and walks the plugin chain. In a stock Kubernetes install the Corefile looks roughly like this:

.:53 {
    errors
    health {
        lameduck 5s
    }
    ready
    kubernetes cluster.local in-addr.arpa ip6.arpa {
        pods insecure
        fallthrough in-addr.arpa ip6.arpa
        ttl 30
    }
    prometheus :9153
    forward . /etc/resolv.conf {
        max_concurrent 1000
    }
    cache 30
    loop
    reload
    loadbalance
}

The kubernetes plugin is authoritative for the cluster zone. It does not run a DNS database. It watches the API server and keeps an in-memory view of Services and EndpointSlices, then synthesises answers on demand. The plugin documentation confirms it watches endpoints through the discovery.EndpointSlices API. The default record TTL is 5 seconds, can be set up to a cap of 3,600 seconds, and 0 disables caching downstream; many distribution Corefiles raise it to 30.

The pods option controls pod A records. disabled (the plugin default) returns NXDOMAIN for them. insecure returns an A record built from the IP in the name without checking that the pod exists, a compatibility behaviour inherited from kube-dns, and the plugin documentation warns it can be abused with wildcard certificates. verified returns a record only if a pod with that IP exists in the namespace, at the cost of watching all pods and therefore more memory.

Anything outside cluster.local falls through to forward, which sends it to the upstream resolvers listed in the node’s /etc/resolv.conf. The cache plugin sits in front of both, with defaults of at most 3,600 seconds for positive answers and 1,800 seconds for denial answers, and a 9,984-entry default capacity (256 shards of 39 items), per the cache plugin documentation. Cache TTLs are always clamped by the record’s own TTL, so a 5-second Service TTL means a 5-second cache entry.

Why the cache does not rescue you from amplification

It is tempting to conclude that caching neutralises the search-list problem. It reduces upstream load, but it does not reduce the packets the pod sends. Every candidate still travels from the pod, through NAT, to a CoreDNS replica, and gets an answer from cache. You save CoreDNS CPU and upstream round trips, not the pod-side latency of the extra hops or the conntrack entries consumed on the node.

There is a second subtlety. The NXDOMAIN answers for api.partner.com.shop.svc.cluster.local are in the cluster zone, so the kubernetes plugin generates them, and they are cached according to the zone’s negative-caching TTL. In a busy cluster these are the most common queries CoreDNS sees, and they are almost always answered from cache or computed from memory. That is why CoreDNS tends to be CPU-bound on packet handling and goroutine scheduling rather than on any real lookup work.

Deeper Analysis: Five-Second Stalls, NodeLocal DNSCache and Discovery Models

The amplification above costs milliseconds. The expensive failures are the ones that cost whole seconds, and they have a precise mechanism in the Linux networking stack. Understanding it explains why NodeLocal DNSCache exists and why it works.

The conntrack race behind the five-second timeout

Most DNS traffic from a pod is UDP, and the destination is a ClusterIP that does not exist on any interface. The kernel’s netfilter layer rewrites it (destination NAT, or DNAT) to a real CoreDNS pod IP and records the translation in the connection tracking table, conntrack, so that the reply can be translated back. Conntrack identifies a flow by its five-tuple: protocol, source IP, source port, destination IP and destination port.

Here is the trap. glibc’s resolver, when it needs both A and AAAA records, sends the two queries in parallel from the same socket, so they share a source port and differ only in payload. The two packets can be processed on different CPUs at nearly the same moment. Both see no existing conntrack entry, both choose the same DNAT target or different targets, and both try to insert an entry. Only one insert succeeds. The other packet is dropped by the kernel and the insert_failed counter in conntrack -S is incremented. The resolver never sees a reply for that query, waits for its timeout (five seconds by default), and retries.

This analysis was published in 2018 in Weave Works’ write-up of racy conntrack and DNS lookup timeouts and mirrored by several other teams, including a widely cited Tencent Cloud container team article. A related variant happens when a reply to a packet that was DNATted arrives from a CoreDNS pod and a stale or conflicting entry exists. Kernel patches that close the races were merged upstream and backported by distributions, around Linux 5.0 for the main same-origin NAT race according to distribution kernel-team mailing list notes, but I could not verify exact version boundaries for every distribution kernel, so check your own kernel changelog before declaring the race fixed.

Figure 2 sequences the main failure.

Figure 2: Two parallel queries from one socket race to create a conntrack entry. The loser is dropped, and the resolver retries only after its timeout.

The mitigations fall into three families. The first changes the resolver: options single-request or single-request-reopen in dnsConfig serialises the A and AAAA queries or uses a new socket for the second, removing the race at the price of extra latency. The musl libc used by Alpine has historically behaved differently (it parallelises and, in older versions, did not honour search-list semantics the same way), which is why Alpine-based images often show different DNS symptoms than Debian-based ones. The second family changes the transport: forcing TCP with use-vc avoids UDP conntrack races at the price of connection setup. The third family removes the NAT from the path entirely, which is what NodeLocal DNSCache does.

NodeLocal DNSCache

NodeLocal DNSCache, stable since Kubernetes v1.18, runs a CoreDNS-based cache as a DaemonSet on every node. Each node-local-dns pod uses host networking, creates a dummy network interface with a link-local address (169.254.20.10/32 by default) and listens there. Pods are configured, via the kubelet’s cluster DNS setting, to send queries to that local address instead of the cluster DNS Service IP.

Because the pod’s destination is now a local address, there is no DNAT and no conntrack entry for the pod-to-cache hop. The node-local cache answers hits immediately. On a miss it forwards to the cluster’s CoreDNS, and the important design choice is that this upstream connection can use TCP rather than UDP. A long-lived TCP connection from each node to CoreDNS replaces thousands of short-lived UDP flows, so conntrack pressure drops, and TCP’s own retransmission replaces the five-second resolver timeout with a much shorter loss-recovery interval. According to the project documentation, the cache’s denial TTL has a 5-second minimum, and a longer value can help performance but may break DNS-polling workflows such as some StatefulSet operators.

Figure 3: With NodeLocal DNSCache, pod queries stay on the node. Misses leave over a persistent TCP connection to CoreDNS, and upstream queries go from CoreDNS to the cloud resolver.

Figure 3 also shows a benefit that matters in clouds: node-local caching reduces the number of packets sent toward the VPC resolver per node. On AWS, the Route 53 Resolver accepts a quota of 1,024 packets per second per network interface for traffic to link-local services, which the AWS VPC documentation describes as an aggregate across DNS queries, instance metadata requests and the Amazon Time Service, and says cannot be increased. When the limit is reached, the resolver rejects traffic. A node running 30 pods with chatty clients and a search list that quadruples every external lookup can approach that ceiling without any single pod looking abusive.

There are operational caveats. The node-local-dns pod is host-networked, so NetworkPolicy namespace selectors may not match its traffic; the project suggests an ipBlock egress rule for the cluster DNS Service IP on port 53 instead. A DaemonSet pod restart makes the local address briefly unavailable on that node, so a rolling update needs care and a sensible maxUnavailable. With IPVS mode kube-proxy or with an eBPF datapath, the interception details differ, so follow the add-on’s matching manifest rather than the generic one. And node-local caching does not reduce the number of packets from the pod: you still emit eight queries for an external name. You merely make each one cheap.

Cutting the amplification at the source

There are four ways to attack the search-list tax, and they compose.

1. Use fully qualified names with a trailing dot in configuration for anything external: api.partner.com. sends one candidate. Many HTTP clients mishandle a trailing dot in the Host header or TLS server name, so test before mass-editing.
2. Lower ndots per pod with dnsConfig.options. Setting ndots:2 means api.partner.com (two dots) is tried as absolute first. In-cluster short names like payments.billing still work because they have fewer than two dots. The risk is a name with two or more dots that you expected to resolve via the search list, for example payments.billing.svc (two dots), which would now be tried as an absolute name first and then fall back, which still works but with an extra failed query.
3. Enable autopath in CoreDNS. The autopath @kubernetes plugin performs the search-list walk server-side. When a query matches the first search suffix, CoreDNS tries each domain in order and returns the first non-NXDOMAIN answer, adding a CNAME from the original name to the real one. The pod sends one query instead of four. The documented caveats are real: it requires pods verified, it maps client IP to pod through a watched cache that can lag (a recycled pod IP can briefly yield the wrong namespace), it is incompatible with Windows nodes, and if the server-side walk ends in NXDOMAIN the client still searches manually.
4. Cache at the application. JVM, Go and Node runtimes have very different DNS caching defaults. A process that resolves on every request amplifies everything above.

A tuned pod spec for an outbound-heavy workload:

spec:
  dnsPolicy: ClusterFirst
  dnsConfig:
    options:
      - name: ndots
        value: "2"
      - name: single-request-reopen
      - name: timeout
        value: "2"
      - name: attempts
        value: "3"

The timeout and attempts options shorten how long a lost packet blocks the caller. They trade resilience to slow upstreams for faster failure, so they are safe where retries exist higher up. Be careful with single-request-reopen: it works around the conntrack race but adds a socket per name, which is overhead you may not need once NodeLocal DNSCache is in place.

Headless Services, SRV records and EndpointSlices

A normal Service returns one ClusterIP and lets the datapath load-balance. A headless Service, declared with clusterIP: None, returns the set of ready pod IPs as multiple A records, and clients choose. This is the basis of StatefulSet identity: with a headless Service and serviceName set, each pod gets a stable name such as kafka-0.kafka-headless.data.svc.cluster.local.

SRV records add port discovery. A query for _client._tcp.kafka-headless.data.svc.cluster.local returns an SRV record per pod with port number and target hostname, which suits protocols that need ports other than the default. Few HTTP clients use SRV, but databases, gRPC resolvers and some service meshes do.

Behind the DNS layer sits the EndpointSlice API. Early Kubernetes used a single Endpoints object per Service, which meant a 5,000-pod Service produced one huge object rewritten on every pod change and shipped to every watcher. EndpointSlices shard that into slices of up to 100 endpoints by default, so churn updates one small object. CoreDNS watches EndpointSlices through the discovery API, so a rolling update of a headless Service changes the DNS answer within the plugin’s watch latency plus the record TTL.

Two details bite in practice. Headless answers only include ready endpoints by default, so a pod that is still starting is invisible, which is usually what you want but surprises operators bootstrapping a clustered database whose peers must discover each other before they are ready; the publishNotReadyAddresses setting exists for that case. And a DNS client that caches an A-record set for the TTL will keep dialling a terminated pod IP, so headless discovery requires clients that re-resolve and handle connection failure.

ExternalName and other Service types

An ExternalName Service returns a CNAME to an external hostname and does no proxying. It is convenient for pointing db.shop.svc.cluster.local at a managed database endpoint, but it brings two sharp edges: no port remapping, and HTTP or TLS clients may send a Host header or SNI value that does not match the target. Because the CNAME is followed by CoreDNS’s forward plugin chain, a slow upstream makes your internal name slow.

DNS-based versus xDS-based service discovery

DNS is one discovery mechanism among several, and its limits are structural. A DNS answer is a snapshot with a TTL. Clients cache, libc resolvers rarely honour small TTLs consistently, and there is no push channel to say “this endpoint is gone”. Health is implicit: only ready endpoints appear, but there is no weight, locality or zone information in a standard A record.

xDS (the discovery-service protocol family used by Envoy, and by Istio and Cilium’s Envoy-based features) inverts the model. The control plane streams endpoint sets, weights, locality and health state to the proxy over a long-lived gRPC connection, and the proxy load-balances per request. Consul offers a hybrid: a DNS interface for simple clients and an HTTP and gRPC API for sophisticated ones. In practice, most meshes still use DNS for the first hop, because the application opens a connection to a name and the sidecar or eBPF layer intercepts it, then use xDS to choose the real endpoint.

Property	Kubernetes DNS (CoreDNS)	Headless Service DNS	xDS (Envoy, Istio)	Consul DNS and API
Update propagation	Pull, TTL bound	Pull, TTL bound	Push, seconds or less	DNS pull, API blocking query
Load balancing	ClusterIP in datapath	Client side, random order	Per request in proxy	Client or proxy
Locality and weights	None	None	Yes	Yes (via metadata)
Per-request cost	One lookup per connection	One lookup per resolve	None at request time	Varies
Failure mode	Stale answers, timeouts	Stale pod IPs	Control plane lag	Agent outage
Client requirement	Any	Re-resolving client	Proxy or mesh-aware	Agent or API client

The pragmatic reading: DNS is the universal fallback and the right default for most traffic; xDS earns its complexity when you need sub-second endpoint convergence, locality-aware routing or per-request policy. For the sidecar-free variant of this design, see our Cilium service mesh deep dive.

Cilium DNS proxy and policy

Cilium adds a different interaction. When an egress policy uses toFQDNs, the agent intercepts DNS responses through a DNS proxy, records the name-to-IP mapping, and uses it to allow traffic to those IPs. That makes DNS an enforcement input, not just a lookup. Two consequences follow. First, DNS latency now sits in the policy path: if the proxy is slow or its cache expires, newly resolved IPs are not yet allowed. Second, your DNS TTL and your application’s cache lifetime interact: an application that holds an IP longer than the policy’s learned entry will see denials. The Cilium external authorization comparison covers how this fits with identity-aware access control.

Operating CoreDNS: Scaling, Observability and a Debugging Workflow

Knowing the mechanisms is half the job. The other half is measuring them in a live cluster, because every DNS problem presents as “the network is slow”.

Sizing and autoscaling CoreDNS

CoreDNS is typically deployed with two replicas, which is a reliability floor and not a capacity plan. The Kubernetes project documents a DNS horizontal autoscaler built on the cluster-proportional-autoscaler, which resizes the CoreDNS Deployment from cluster size rather than from query load. In its linear mode the replica count is the larger of ceil(cores / coresPerReplica) and ceil(nodes / nodesPerReplica), with optional minimum and maximum bounds. Cloud providers ship variants: GKE’s kube-dns autoscaler and the managed-add-on equivalents in other clouds use the same idea.

Linear autoscaling by node count is a proxy for query rate and nothing more. A cluster of 20 nodes running a batch workload that makes a million external lookups an hour needs more DNS capacity than 100 idle nodes. The better signal is the rate and latency metrics CoreDNS exports. If you scale on query rate, a HorizontalPodAutoscaler fed from coredns_dns_requests_total is a reasonable design, and you should add a PodDisruptionBudget so a node drain cannot remove every replica.

Capacity per replica is workload-specific and I will not quote a throughput figure, because it depends on cache hit rate, upstream latency, response sizes and CPU. Measure it: run a load generator such as dnsperf against a single replica with a realistic query mix, record the rate at which p99 latency starts to climb, and keep production at 50 to 60 percent of that. Also set CPU requests, not just limits. A CPU-throttled CoreDNS pod (cgroup CFS throttling) looks identical to a slow upstream from the client side.

Anti-affinity matters as much as replica count. Spread CoreDNS across nodes and zones so that a single node failure takes out at most one replica, and keep the lameduck 5s setting in the health plugin so a terminating replica finishes in-flight queries instead of dropping them.

The metrics that tell you what is wrong

With the prometheus plugin enabled (port 9153 in the default Corefile), these series answer most questions:

• coredns_dns_requests_total and coredns_dns_responses_total by response code. A high share of NXDOMAIN is the ndots signature. In a healthy cluster with default ndots:5, expect NXDOMAIN to be a large fraction of traffic; a sudden jump means a client started looking up a new external name or a broken configuration.
• coredns_dns_request_duration_seconds histogram, for latency percentiles inside CoreDNS. If this is low while clients see seconds of delay, the problem is before CoreDNS (conntrack, NAT, packet loss), which is exactly the five-second signature.
• coredns_cache_hits_total, coredns_cache_requests_total and coredns_cache_entries, for cache effectiveness and capacity pressure. If entries sit at capacity and coredns_cache_evictions_total climbs, the default 9,984-entry capacity may be too small for your name diversity.
• coredns_forward_requests_total and coredns_forward_healthcheck_failures_total for upstream health.

On the node, conntrack -S shows insert_failed and drop counters per CPU, and conntrack -C gives the current entry count to compare against nf_conntrack_max. A rising insert_failed count during DNS stalls is strong evidence of the race.

A debugging runbook

Work from the pod outward. Each step rules out one layer.

Figure 4: A layer-by-layer DNS debugging flow. Each branch isolates the pod resolver, the NAT path, CoreDNS itself or the upstream.

Step 1: check the pod’s resolver view. Exec into an affected pod and read /etc/resolv.conf. Confirm the nameserver is the cluster DNS Service (or the node-local address), that ndots is what you think it is, and that the search list is not truncated.

Step 2: separate name types. Run dig +search +showsearch or getent hosts to reproduce what the application’s resolver does, then dig @10.96.0.10 payments.shop.svc.cluster.local (use your cluster’s DNS IP) to talk to CoreDNS directly. If direct queries are fast and the application is slow, the culprit is the search list or the application’s resolver, not CoreDNS.

Step 3: time the failure. Intermittent delays of almost exactly five seconds point at a lost UDP packet and a resolver retry, which means the conntrack race or packet loss. Delays that scale with load and with no fixed value point at CoreDNS or upstream saturation. Run a loop of queries and record the distribution, for example for i in $(seq 1 200); do dig +tries=1 +time=6 payments.shop.svc.cluster.local | grep 'Query time'; done.

Step 4: capture packets. On the node, tcpdump -ni any port 53 while reproducing shows what really happens. Look for queries with no reply, retransmissions spaced five seconds apart, and the order of A and AAAA queries from the same source port. If you see the pair leave and only one reply return, you have the race.

Step 5: inspect CoreDNS. Enable the log plugin briefly (it is verbose and costly at scale), or read the metrics above. Check for SERVFAIL and for forward errors. A loop plugin crash on startup indicates a forwarding loop, commonly because the node’s /etc/resolv.conf points at 127.0.0.53 (systemd-resolved) and CoreDNS forwards to itself.

Step 6: check the upstream and the cloud. If CoreDNS metrics show slow forwards, test the upstream resolver directly from a node. On AWS, count the DNS queries per second leaving each node’s network interface and compare with the 1,024 packets per second link-local quota. AWS documents that when you reach it the Route 53 Resolver rejects traffic, and Amazon publishes CloudWatch metrics for the resolver on instances that support them (verify the metric name and availability for your instance type in the current EC2 documentation).

Trade-offs, Gotchas, and What Goes Wrong

No fix is free. NodeLocal DNSCache adds a DaemonSet that becomes critical node infrastructure; if it is down, pods that were configured to use the link-local address have no DNS on that node. The project’s design keeps the cluster DNS Service as an alternate path in some setups, but you should test failure explicitly by killing the pod and watching a client.

Negative caching can hurt as much as it helps. A cached NXDOMAIN for a Service that does not exist yet persists for the denial TTL. A deployment pipeline that creates a Service and then immediately starts pods that look it up can see NXDOMAIN for the cache lifetime. The kubernetes plugin TTL, CoreDNS cache denial settings and the node-local cache each have their own value; the effective lifetime is determined by the chain, and tuning one without the others yields surprises. Raising the cache denial TTL is attractive for performance, and the documentation specifically warns that it may break workflows that poll DNS, such as StatefulSet operators waiting for peers to appear.

autopath shifts load rather than removing it. The server-side walk makes CoreDNS do the work of several lookups per query and needs the pod cache from pods verified, which costs memory proportional to pod count. In clusters with high pod churn the IP-to-pod map can briefly lag, which the autopath documentation calls out as a source of wrong-namespace answers.

Lowering ndots breaks assumptions. Some applications and Helm charts use names such as redis.cache.svc relying on the search list. Changing the default globally is risky; apply it per workload and test with the real clients.

Retries multiply DNS load. A resolver that times out and retries three times against a slow CoreDNS adds load exactly when CoreDNS is weak, the same retry-storm dynamic we analysed in our post on retry budgets and circuit breakers. Keep attempts modest, spread replicas, and cache aggressively at the node.

TCP fallback is not free either. Large responses (many pod IPs in a headless Service) can exceed the 512-byte classic UDP limit or the EDNS buffer size and force a TCP retry. Under high rates this opens many short TCP connections through conntrack. Large headless Services are better served by SRV with a bounded set or by a discovery mechanism that is not DNS.

Hidden dependencies on node DNS. The forward . /etc/resolv.conf default means every cluster inherits the node’s upstream configuration. A node image change that swaps resolvers, or a DHCP lease that updates search domains, alters CoreDNS behaviour cluster-wide with no Kubernetes change.

Service meshes and ingress add layers. A mesh sidecar may resolve names itself, skipping the pod’s ndots path, or capture DNS through an interception rule. When you migrate edge traffic, such as in the ingress-nginx retirement and Gateway API migration playbook, confirm which component now performs name resolution for upstreams, because the new data plane may use different resolver settings and caching.

Practical Recommendations

Start by measuring, because the right fix depends on which failure you have. Pull the NXDOMAIN ratio and request rate from CoreDNS metrics, check conntrack -S for insert_failed on busy nodes, and count node-level DNS packets per second against your cloud’s limit. Those three numbers tell you whether you have an amplification problem, a race problem or an upstream throttling problem.

For most production clusters the ordering that pays off is: deploy NodeLocal DNSCache first, because it removes the conntrack race and the packet-rate pressure with no application changes; then reduce ndots for outbound-heavy workloads and use trailing dots for hot external names; then consider autopath only if you still see heavy search-list traffic and can afford the memory and caveats. Fix application DNS caching last, but do not skip it, since a runtime that resolves on every request defeats every layer below it.

Treat CoreDNS as a tier-one service: set CPU requests, a PodDisruptionBudget, anti-affinity across zones, and alerts on latency and SERVFAIL rate. Test failure by terminating a CoreDNS pod and a node-local cache pod during a load run.

Checklist:

• [ ] CoreDNS metrics scraped; alerts on p99 latency, SERVFAIL ratio and NXDOMAIN ratio change.
• [ ] NodeLocal DNSCache deployed and verified (pods point at the link-local address; kill-test passed).
• [ ] conntrack -S insert_failed tracked on nodes; kernel changelog checked for conntrack race fixes.
• [ ] ndots reviewed per workload; trailing dots or FQDNs used for hot external names.
• [ ] CoreDNS replicas sized from measured load, spread across zones, with a PodDisruptionBudget.
• [ ] Cloud resolver packet quota (for example 1,024 packets per second on AWS link-local services) monitored per node.
• [ ] Application DNS caching behaviour documented for each runtime in use.
• [ ] Negative-cache TTLs reviewed against deployment ordering and StatefulSet bootstrap.

Frequently Asked Questions

Why does Kubernetes use ndots:5 by default?

Because in-cluster names are long and developers want short ones. With ndots:5, payments, payments.shop and payments.shop.svc all work through the search list, since each has fewer than five dots. The cost is that external names with fewer than five dots, which is nearly all of them, are tried against each search suffix first. You can override it per pod with dnsConfig.options, or avoid it with a trailing dot.

How many DNS queries does one external lookup generate in Kubernetes?

With the default three Kubernetes search domains and ndots:5, a name like api.partner.com produces four candidates: three search-suffixed names that return NXDOMAIN, then the real name. If the resolver sends both A and AAAA for each, that is eight queries for one lookup. Node-provided search domains add more. A trailing dot or a lower ndots brings it down to two queries.

What causes the 5 second DNS delay in Kubernetes?

Usually a lost UDP packet. When the resolver sends A and AAAA queries in parallel from the same socket, a race in the kernel’s connection tracking can cause one packet to be dropped during NAT, and the resolver retries after its five-second default timeout. NodeLocal DNSCache, single-request-reopen, TCP for upstream queries, and kernel fixes for the race all reduce it.

Do I need NodeLocal DNSCache?

Not for every cluster, but it is the highest-value change for busy ones. It is stable since Kubernetes v1.18 and gives you local cache hits, no NAT for the pod-to-cache hop, TCP to CoreDNS, and fewer packets toward your cloud resolver. Small, low-traffic clusters may never notice the problems it solves, and it adds a node-critical DaemonSet to operate.

What is the difference between a headless Service and a normal Service for DNS?

A normal Service returns one stable ClusterIP and the datapath load-balances across ready pods. A headless Service (clusterIP: None) returns the ready pod IPs directly, so clients pick an endpoint and must re-resolve to see changes. It also gives StatefulSet pods stable per-pod DNS names, and supports SRV records for named ports.

Should I use DNS or xDS for service discovery?

Use DNS as the default: it is universal, simple and good enough for most services. Choose xDS through a mesh or Envoy when you need fast endpoint updates, locality-aware routing, weighted load balancing or per-request policy. Many production systems use both, with DNS for the first hop and xDS to select the final endpoint.

Further Reading

• CNI comparison: Calico, Cilium, Flannel and Multus for the datapath that carries DNS traffic.
• Cilium external authorization versus OAuth2 Proxy and Istio for identity-aware access on top of discovery.
• Cilium service mesh, sidecarless and eBPF deep dive for DNS-aware policy and xDS in practice.
• Ingress-nginx retirement and Gateway API migration playbook for edge changes that alter upstream resolution.
• Kubernetes documentation: DNS for Services and Pods
• Kubernetes documentation: Using NodeLocal DNSCache
• CoreDNS kubernetes plugin reference and CoreDNS cache plugin reference
• Amazon VPC documentation: Understanding Amazon DNS

By Riju — about