Skip to content

We enabled IPv6. Why did only some customers lose the API?

The Route Exists. Does the Request Arrive? · Part 5 of 6 · 5 min read

Read the worked explanation ↓

Interview question. An API has served IPv4 clients for years. The team adds an AAAA record and dual-stack listeners. Most users are unaffected, but customers on a few mobile networks now see intermittent connection failures and long delays. IPv4 health checks remain green. Explain dual-stack connection selection, how to distinguish broken IPv6 from application failures, and how to deploy safely.

Start with a small correction to the incident statement. Adding an AAAA record doesn't command every client to use IPv6. It gives clients an IPv6 destination address to consider alongside an A record containing IPv4 addresses. The operating system and application choose which addresses to try, based on configuration, reachability history and their connection-establishment algorithms. Some libraries try addresses serially. Many modern clients use variants of Happy Eyeballs to avoid waiting too long on a broken address family.

If a client attempts the IPv6 destination first and the path silently drops SYN packets, a naive implementation can wait for a retransmission timeout before attempting IPv4. Other clients quickly race or stagger IPv4 attempts. That explains why the same rollout can be invisible to one app and painful to another without implying the IPv6 address is universally unreachable.

A dual-stack client resolves A and AAAA records, attempts IPv6, then races IPv4 when the first path fails, illustrating divergent client outcomes.
A dual-stack client resolves A and AAAA records, attempts IPv6, then races IPv4 when the first path fails, illustrating divergent client outcomes.
Think first

Your CDN publishes AAAA for an API. Error rate rises only for two mobile carriers, while IPv4 synthetic tests remain successful. Propose a reproducible test matrix and a reversible launch plan that distinguishes DNS, address selection, IPv6 routes, ICMPv6, TLS and gateway policy.

Take a few minutes to form an approach, then compare it with the worked explanation.

Reveal the worked explanation

Follow both address families independently

Suppose api.example.com now has an A record and an AAAA record. Run controlled probes over forced IPv4 and forced IPv6 from affected access networks. Can both establish TCP, complete TLS and fetch the same authenticated endpoint? If IPv6 connects but the response stalls, inspect the return path, server bind configuration, routing, firewall policy and path MTU. IPv6 relies on ICMPv6 for essential functions, including Packet Too Big signaling. Blindly blocking all ICMPv6 can produce failures that look like application hangs.

A record existing in DNS is not evidence that your infrastructure is correctly reachable through that address family. Your public load balancer might listen on IPv6 while its security policy permits the wrong source ranges. A proxy might pass through the connection but an application allowlist might interpret IPv6-mapped or native IPv6 addresses incorrectly. On a mobile carrier, an IPv6-only access network can also use NAT64 and DNS64 to reach an IPv4-only origin. Hard-coded IPv4 literals or assumptions that every client has native IPv4 can fail on these networks.

The first diagnostic output I want is the breakdown of connection attempts and outcomes by address family. If application logs only record successful requests, they may hide all the connections that never reached the process. Add measurements from edge listeners and external probes: DNS result, selected destination family, connect outcome, TLS outcome, handshake time and fallback time. Group failures by carrier or source autonomous system rather than relying on a single office network test.

What Happy Eyeballs really promises

RFC 8305 describes Happy Eyeballs version 2. It resolves both families asynchronously, orders available addresses and starts connection attempts with a stagger so a nonworking choice doesn't stall the whole connection indefinitely. The RFC recommends an initial connection-attempt spacing around 250 milliseconds in a simple configuration, but that number is not a universal browser setting or a guaranteed fallback time. Implementations vary, historical RTT data can influence decisions, and application frameworks can behave differently.

Racing also does not make an unhealthy IPv6 service acceptable. At high request volumes, even a short fallback delay or extra connection attempt can affect user-perceived p99, radio power use and network traffic. Some clients are on IPv6-only networks and cannot rely on a direct IPv4 fallback at all. Happy Eyeballs improves connection establishment. It is not an availability substitute for operating both address families properly.

The overlooked policy boundary

Enabling IPv6 changes more than the socket type. IPv4 rate limits sometimes count source a.b.c.d addresses as client identities. With IPv6, customers may rotate privacy addresses inside a prefix. Rate-limiting every /128 address separately can allow a single actor to spread requests across many addresses. Blocking an entire /64 without considering shared networks and provider assignment can penalize legitimate users. The correct aggregation depends on the threat model and authentic user or device identity, not an arbitrary prefix constant.

Audit firewalls, WAF rules, monitoring dashboards, geo controls, access logs and TLS origin policies for both families. A service that processes the IPv4 path through an API gateway but sends IPv6 directly to an origin could accidentally bypass authentication or request limits. Check the entire policy chain before treating a successful IPv6 handshake as deployment approval.

Roll out with evidence, not faith

I would initially expose IPv6 to test domains and controlled clients, including IPv6-only and NAT64 networks. Then deploy AAAA gradually where the DNS and routing setup allows segmentation. Compare forced-family connection success, TLS success, end-to-end latency and fallback counts by region and carrier. If the canary fails, remove the new AAAA announcement where appropriate, but remember that cached DNS answers and existing connections won't disappear immediately. Keep the affected listener ready through the drainage period instead of assuming the DNS change instantaneously rolls back every client.

The interview answer is not simply "implement Happy Eyeballs." It is to separate name resolution, address-family selection, network reachability, client fallback and security policy. A new IPv6 address creates a new path through all of those controls. A reliable rollout proves that path works for users who cannot or do not take the old one.

Go deeper

RFC 8305 supersedes the original Happy Eyeballs description and recommends staggered connection attempts rather than waiting for all DNS responses. The default delay example is 250 ms, subject to other timing rules and client variation. RFC 6724 covers address selection. IPv6 Path MTU Discovery needs appropriate ICMPv6 behavior. DNS64/NAT64 can let IPv6-only clients reach IPv4 services in supported environments, but IPv4 literals and incompatible client behavior can create failures. Observe forced IPv4 and IPv6 probes rather than inferring family success from general request metrics.

Check your reasoning

  • Separate A and AAAA DNS answers from actual client address choice.
  • Explain Happy Eyeballs staggered connection attempts and implementation variation.
  • Test IPv4 and IPv6 paths separately including TLS, firewall and ICMPv6 PMTU.
  • Handle IPv6-only clients, DNS64/NAT64 and IPv4-literal limitations.
  • Audit rate limiting, logs and rollout cohorts across both address families.

Common mistake: Every dual-stack client will fall back to IPv4 within exactly 250 ms if IPv6 is broken.

A dual-stack rollout works on IPv4 but stalls for some customers. Which validation best identifies the failing path?

Read the answer explanation

Force-family test affected carriers and review address selection, route and policy parity — RFC 8305 provides an algorithm and recommended delays, not a universal fallback guarantee. Some clients are IPv6-only.

Apply it elsewhere

You run abuse controls that bucket clients by IPv4 /32 addresses. Extend the design to native IPv6 clients without either allowing trivial address rotation to bypass controls or blocking entire shared networks by accident.

Compare your approach

Explain asynchronous address-family resolution, staggered attempts, non-universal timing, IPv6-only and NAT64 constraints, route and ICMPv6 health, policy parity and canary measurement.

One more constraint

Does publishing both A and AAAA records guarantee every client will successfully fall back to IPv4 if the IPv6 path is unusable?

Quick revision

Dual-stack DNS returns both IPv4 and IPv6 destinations, but clients choose and attempt them differently. A broken IPv6 route or policy can cause timeouts for clients that prefer that family, while Happy Eyeballs implementations may stagger an IPv4 attempt to reduce delay. A fallback is not guaranteed for IPv6-only users. Test connection and TLS outcomes separately by address family and carrier, audit IPv6 firewall and rate-limit controls, and roll out AAAA with measured canaries and reversible drainage.

References