Skip to content

The API is healthy. Why does the first request wait on DNS?

The Connection Is More Than an IP · Part 1 of 6 · 5 min read

Read the worked explanation ↓

Interview question. A mobile checkout app spends 120 milliseconds resolving api.example.com on its first request, then resolves the same name almost instantly for several minutes. A developer says DNS is one server returning one IP and suggests increasing the API instance count. Trace what actually happens, including caches, recursive and authoritative resolvers, and explain where the time can come from.

Start with a question the developer has skipped: who is answering the DNS query? The browser may already know the address. The operating system may have a cached result. A local stub resolver may forward to a corporate or ISP recursive resolver, which may already have the relevant records. A warm lookup can finish without a packet ever reaching the DNS authority for example.com. A cold lookup might need several exchanges with other nameservers. Neither situation tells us whether an API server is slow.

An application asks for address records such as A and AAAA. On a traditional recursive lookup, its configured recursive resolver does the discovery work. When the answer is not cached, that resolver can contact a root nameserver, receive a referral toward .com, ask a .com nameserver where example.com is delegated, and finally ask an authoritative nameserver for the relevant record. The root does not carry the final IP address for every hostname. It tells the resolver where to look next.

A client asks a recursive resolver, which follows referrals from root through .com to the authoritative zone and caches the final answer.
A client asks a recursive resolver, which follows referrals from root through .com to the authoritative zone and caches the final answer.
Think first

An API moves behind a CNAME pointing to a CDN provider. A cold lookup takes 170 ms in one region and 12 ms in another. Outline how you would isolate stub cache, recursive cache, CNAME chain, nameserver referral and actual TCP connection time, without assuming the CDN is at fault.

Take a few minutes to form an approach, then compare it with the worked explanation.

Reveal the worked explanation

A recursive resolver is doing two different jobs

To the application, the recursive resolver looks like a service that returns an answer or an error. Behind that interface it might behave as an iterative client, following referrals and querying multiple candidate nameservers. There may be more steps when DNSSEC validation, CNAME chains, address resolution for nameservers, or network retries are involved. The simplified root, TLD, authority path is the mental model, not a promise of exactly three packets.

The recursive resolver also caches record sets. Suppose the authoritative A record has a TTL of 300 seconds. A resolver that fetches it may reuse that result while its remaining TTL is positive. If another client asks 120 seconds later, the recursive resolver might return it with roughly 180 seconds remaining. The client is not guaranteed a fresh five-minute cache window starting from its own request. A second cache layer can make the observed timing more complicated.

This is why a packet capture from the mobile application often shows one question and one answer. It does not show the recursive resolver's earlier work. On the other hand, issuing dig directly against an authoritative server bypasses much of the path the actual application uses. Both measurements are valid, but they answer different questions.

The authoritative service is responsible for the records of the zone it serves. The recursive resolver is responsible for trying to obtain answers on behalf of clients. Confusing these roles leads to fixes in the wrong place. Adding API instances will not shorten a recursive lookup unless the change also affects the network path or endpoint selection in a relevant way.

An answer can be more than an address

An alias can appear in the chain. api.example.com might return a CNAME naming a provider-controlled hostname, and that target requires its own address lookup. The two record sets have their own TTLs and potentially different authorities. The resolver must follow the chain subject to normal DNS rules. An HTTPS resource record can additionally advertise connection hints for capable clients, including alternatives for HTTP protocols. None of this means DNS itself establishes the TLS connection or decides application authorization.

Negative answers can be cached too. If a deployment queries preview.api.example.com before the name exists and later creates it, some resolvers can continue returning the earlier authoritative negative response until its negative caching interval expires. A retry storm against the application's own DNS client does not manufacture the missing positive answer. The negative caching behavior is tied to information in the zone's SOA record, not simply the new A record's TTL.

Investigate the right stage

For the original 120-millisecond report I would split the timeline into DNS, connection establishment, TLS, and request processing. In the client I want cold and warm measurements, resolver identity when observable, address family, and whether the request reused an established connection. In controlled testing I would compare the configured recursive resolver with direct authoritative queries, but I would not call their timings interchangeable.

If only one recursive service is slow, examine its cache misses, upstream reachability and retry patterns. If the authoritative service responds slowly across independent resolvers, inspect authoritative latency and delegation health. If the client alternates between IPv4 and IPv6 results, examine address-family reachability as well as the DNS time. If DNS time is low but connection time is high, leave the DNS service alone and investigate the transport path.

A 300-second TTL is not a promise that the client asks every five minutes. Browsers, runtimes, operating systems and recursive resolvers have their own caching rules, prefetching and connection reuse. Some also provide controlled stale answers during an authority outage. Before treating a latency chart as one universal lookup path, determine which resolver and which cache produced the measurement.

The design decision is not whether to make DNS infinitely fast. It is where to keep authoritative records correct, how to make recursive lookups reliable, and how to observe the separate stages. The next incident follows naturally: the DNS answer changes during an IP migration, and not everybody hears about it at the same time.

Go deeper

RFC 1034 and RFC 1035 describe the DNS hierarchy, resolvers, record classes and TTL behavior. A recursive server may cache referrals, address data and final answers independently. A cache hit does not require repeating every step of the discovery path. RFC 2308 describes negative caching for NXDOMAIN and NODATA. RFC 8767 defines constrained stale-answer behavior for recursive resolvers when refresh fails. A DNSSEC-validating resolver also performs cryptographic validation rather than simply trusting any address returned by an upstream source. The client-facing DNS protocol does not guarantee that one name maps to one address or that all clients see records in the same order.

Check your reasoning

  • Separate stub, recursive and authoritative resolver responsibilities.
  • Trace cold root/TLD/authority referrals without claiming a fixed packet count.
  • Explain record TTL, remaining TTL and ordinary cache reuse.
  • Include CNAME, negative caching and optional stale answers as separate behavior.
  • Design client and resolver stage-specific latency measurements.

Common mistake: A fresh browser request necessarily contacts the authoritative DNS server.

Cold DNS lookup is slow while warm lookup is fast and API execution is healthy. Which investigation targets the right boundary?

Read the answer explanation

Separate client cache, recursive cache and authoritative latency measurements — A recursive cache hit can answer locally. Slow cold resolution needs stage-by-stage diagnosis.

Apply it elsewhere

A service-discovery name is created after a staging workload already queried it and received NXDOMAIN. New workers still report no address. Explain what you would measure and which caching intervals matter before changing the deployment controller.

Compare your approach

Separate stub, recursive and authoritative roles, explain cold referrals, account for record and negative TTLs, distinguish DNS work from TLS and TCP work, and propose measurements at each stage.

One more constraint

A recursive resolver answers a cached A record with 40 seconds remaining. Does that client necessarily hold the result for the authoritative record's original 300 seconds?

Quick revision

DNS resolution is a hierarchy, not one lookup server. An application commonly asks a recursive resolver, which may answer from cache or follow root, TLD and authoritative referrals to obtain records. TTLs govern ordinary reuse at different caches, while negative answers can also persist. CNAMEs and DNSSEC can add work. Separate client cache, recursive cache, authority, TCP and TLS timing before assigning blame. A warm result does not prove the authority was queried again.

References