DNS records, recursive resolution, TTLs and caching, how .NET resolves names, connection pooling vs DNS changes, Kubernetes ndots and conntrack issues, and diagnosis tools.
DNS maps names to records: A (IPv4), AAAA (IPv6), CNAME (alias), plus MX, TXT, SRV, NS and PTR. A client asks a recursive resolver, which walks root → TLD → authoritative servers and caches the answer for its TTL. .NET uses the OS resolver (getaddrinfo) and does not cache itself.
Typical incidents:
HttpClient kept pooled connections to the old IP. Fix with PooledConnectionLifetime.ndots:5 makes each external name try several search domains first, and UDP conntrack races cause occasional 5-second delays.Diagnose with dig/nslookup from the same pod, check TTLs, and look at the DNS phase of the HTTP timing.
app ──► OS resolver (cache, /etc/hosts, /etc/resolv.conf)
│
▼
recursive resolver (CoreDNS in k8s, Azure DNS 168.63.129.16, VPC resolver)
│ cache miss
├──► root servers ── "ask .com"
├──► .com TLD ── "ask ns1.example.com"
└──► authoritative ── api.example.com CNAME api-prod.trafficmanager.net
api-prod... A 203.0.113.10 TTL 60
| Record | Purpose |
|---|---|
| A / AAAA | Name to IPv4 / IPv6 |
| CNAME | Alias to another name (cannot sit at the zone apex) |
| MX | Mail servers |
| TXT | Verification, SPF, DKIM |
| SRV | Service host and port |
| NS | Delegation to name servers |
| PTR | Reverse lookup (IP to name) |
An AKS service calling a SaaS API had about 1% of requests taking just over 5 seconds, always in the DNS phase. Two causes combined:
/etc/resolv.conf in pods had ndots:5 and search domains ns.svc.cluster.local svc.cluster.local cluster.local. api.vendor.com has two dots, so the resolver first tried api.vendor.com.ns.svc.cluster.local and others: up to 8 queries (A and AAAA) per lookup.Fixes: call the external host with a trailing dot (api.vendor.com.) or lower ndots to 2 in the pod dnsConfig, enable NodeLocal DNSCache, and reuse connections so lookups happen rarely. The 5-second tail disappeared.
Probes DNS knowledge beyond "turns names into IPs" and the ability to debug production DNS issues.
Green flags
Red flags
authoritative TTL 60 s
│
├─ recursive resolver caches ≤ 60 s (some clamp to a minimum)
├─ OS cache (systemd-resolved, Windows DNS Client) caches ≤ 60 s
└─ YOUR APP: HttpClient connection pool keeps the TCP connection
to the old IP for as long as it lives ← often the real "cache"
Negative caching: "name does not exist" (NXDOMAIN) is also cached, based on the zone's SOA settings. Creating a record right after a failed lookup can still fail for a while.
Dns.GetHostAddressesAsync and SocketsHttpHandler call the OS resolver. There is no long-lived DNS cache in .NET itself.HttpClient resolves only when it opens a new connection. A long-lived pooled connection never re-resolves, so set PooledConnectionLifetime (e.g. 1–5 min) or use IHttpClientFactory.AAAA lookups are slow or broken, every connection pays for them; check with dig AAAA.System.Net.NameResolution counters/meters show lookup counts and durations.orders (same namespace), orders.payments (other namespace), orders.payments.svc.cluster.local (FQDN).ndots:5: names with fewer than 5 dots try search domains first. For external names this multiplies queries.dnsConfig.options: ndots: "2", NodeLocal DNSCache (also avoids the conntrack UDP race), scale CoreDNS.mydb.database.windows.net resolves to a private IP inside the VNet. Missing zone links mean the app resolves the public IP and is blocked by the firewall.cat /etc/resolv.conf # nameserver, search, ndots
dig api.vendor.com +search +stats # what the pod actually resolves
dig api.vendor.com @8.8.8.8 # compare with a public resolver
dig +trace api.vendor.com # walk the delegation
nslookup mydb.database.windows.net # private vs public IP?
kubectl -n kube-system logs deploy/coredns # errors, timeouts
cat /etc/resolv.conf, then dig +search +stats example.com and count queries in CoreDNS logs (enable log plugin in a test cluster).example.com. (trailing dot).PooledConnectionLifetime expires.// Make DNS changes take effect: bounded connection lifetime
builder.Services.AddHttpClient<VendorClient>(c => c.BaseAddress = new Uri("https://api.vendor.com/"))
.ConfigurePrimaryHttpMessageHandler(() => new SocketsHttpHandler
{
PooledConnectionLifetime = TimeSpan.FromMinutes(2), // re-resolve at least every 2 minutes
ConnectTimeout = TimeSpan.FromSeconds(5)
});
// Prefer fixing search-domain fan-out in the pod's dnsConfig (below) rather than putting a
// trailing dot in URLs: a trailing dot also ends up in the Host header and SNI, which some
// servers and certificates reject.
// Quick diagnostic endpoint (internal only): what does this pod resolve, and how fast?
app.MapGet("/diag/dns/{host}", async (string host, CancellationToken ct) =>
{
var sw = Stopwatch.StartNew();
var addresses = await Dns.GetHostAddressesAsync(host, ct);
return Results.Ok(new { host, ms = sw.ElapsedMilliseconds, addresses = addresses.Select(a => a.ToString()) });
}).RequireAuthorization("Ops");
# Pod DNS tuning
spec:
dnsConfig:
options:
- name: ndots
value: "2"
- name: single-request-reopen # glibc: avoid parallel A/AAAA on one socket
Mistake: Assuming a DNS change reaches running services once the TTL expires.
Why it is wrong: Pooled HttpClient connections keep talking to the old IP for as long as they live, regardless of TTL.
Instead: Set PooledConnectionLifetime (for example two minutes) or use IHttpClientFactory.
Mistake: Ignoring ndots and search domains in Kubernetes.
Why it is wrong: External names with fewer than five dots try every search domain first, multiplying queries and latency.
Instead: Lower ndots to 2 in the pod's dnsConfig and enable NodeLocal DNSCache. Avoid putting a trailing dot in request URLs, because it also ends up in the Host header and SNI, which some servers and certificates reject.
Mistake: Testing DNS from your laptop instead of from the failing pod or VNet.
Why it is wrong: Split-horizon and private DNS zones return different answers in different networks.
Instead: Run dig or nslookup from the same pod or subnet as the app.
Mistake: Setting very long TTLs on records used for failover. Why it is wrong: Resolvers and clients keep the old address for hours, so failover is slow. Instead: Use 30 to 300 second TTLs for records that may move and longer ones for stable records.
Mistake: Blaming the remote API for 5-second spikes. Why it is wrong: Exactly-5-second outliers are a classic resolver timeout and retry on your side. Instead: Split timing into DNS, connect, TLS and server time before escalating.
"It's always DNS" is a joke because DNS sits under everything and fails in subtle ways: stale answers, slow answers, and different answers depending on where you ask. A senior engineer understands the interplay between resolution (OS resolver, CoreDNS, cloud resolver, TTLs) and connection management (the HttpClient pool), because in .NET the pool is usually what holds on to old IPs.
| Symptom | Likely cause | Fix |
|---|---|---|
| Traffic still hits old IP after failover | Long-lived pooled connections | PooledConnectionLifetime, IHttpClientFactory |
| 5-second tail latency | Resolver timeout (conntrack race) | NodeLocal DNSCache, single-request-reopen, lower ndots |
| High CoreDNS load | ndots:5 fan-out for external names |
Lower ndots, cache locally |
| Private endpoint blocked by firewall | Name resolves to public IP | Link the private DNS zone to the VNet |
| Works in one namespace only | Short name resolved via search path | Use service.namespace or the FQDN in config |
Service meshes add another layer (DNS proxying, routing by service identity), and DNSSEC protects integrity only when the resolver validates it. I prefer platform fixes over code workarounds and document the expected resolution for critical dependencies so drift is detectable.
A load balancer spreads traffic across healthy instances.
Health checks remove unhealthy instances (in Kubernetes: readiness for traffic, liveness for restarts). Algorithms: round robin, least connections/requests, hashing. Sticky sessions pin a user to one instance, which hurts scaling and failover. Connection draining stops new traffic to an instance before shutdown so in-flight requests finish.
gRPC uses long-lived HTTP/2 connections, so an L4 balancer balances connections, not calls: one client connection carries all its calls to one pod. Use L7 balancing (mesh/Envoy) or client-side load balancing.
Two different limits produce similar errors:
EMFILE, "Too many open files"): the process hit its ulimit -n. Something is leaking files or sockets.EADDRNOTAVAIL, "Cannot assign requested address"): too many outbound connections to the same destination, usually closed sockets stuck in TIME_WAIT for 60 s.The classic .NET cause is new HttpClient() per request. Fix it with a shared HttpClient using SocketsHttpHandler { PooledConnectionLifetime = ... }, or IHttpClientFactory. Diagnose with ls /proc/<pid>/fd | wc -l, lsof -p, ss -s and ss -tan state time-wait.
A firewall allows or blocks traffic based on rules (source, destination, port, protocol, and for L7 firewalls/WAFs, HTTP content).
Troubleshoot by symptom: timeout usually means dropped by a firewall or routing, refused means reached the host but nothing listening, reset after idle often means a firewall or load balancer idle timeout removed the connection state. Test from the source (nc -vz, Test-NetConnection), check rules and flow logs in both directions, and set TCP keep-alives or shorter pool idle timeouts below the firewall's idle timeout.
How to tell it (STAR)
**S:** Our [payments service] had [1%] of vendor calls taking just over five seconds, and the vendor said they were fine. **T:** I was asked to find the cause before [peak season]. **A:** Tracing showed the delay was in DNS. In the pod, `ndots:5` turned each lookup into [8] queries, and a known conntrack race occasionally dropped one, triggering the 5-second resolver timeout. I used the fully qualified name, lowered ndots, enabled NodeLocal DNSCache and reused connections longer. **R:** The 5-second tail disappeared and CoreDNS load dropped by [70%].
Last updated 23 Oct 2026.