Skip to content

The IP address stayed the same. Why did the connection move to another city?

The Route Exists. Does the Request Arrive? · Part 4 of 6 · 5 min read

Read the worked explanation ↓

Interview question. A global DNS and CDN provider serves clients through the same public address from multiple cities. A customer establishes a long-lived connection through one city. During a network maintenance event, the route changes. Some connections reset even though the destination IP does not change. Explain anycast, what actually selects the site, and how to drain a location without pretending routing changes preserve application state.

Most engineers learn that an IP address identifies one machine or one load balancer. Anycast breaks that useful simplification on purpose. Multiple service locations advertise reachability to the same destination address or prefix. The network routes a client's packets to one of those locations according to its current routing information and policies. No global central scheduler is examining every HTTP request and selecting the physically closest server.

Take a public resolver reachable at an anycast address. A customer in Singapore may reach an edge in Singapore while another customer in London reaches a European edge, even though both send packets to the same IP address. Those are routing outcomes, not separate A records for the two cities. A geographic label is at best a clue. BGP path policy, peering, topology and changing reachability can send a customer to a site that is not physically nearest.

Two edge sites announce one service IP. A routing change moves a client's packets from the first site to the second, which lacks the original TCP connection state.
Two edge sites announce one service IP. A routing change moves a client's packets from the first site to the second, which lacks the original TCP connection state.
Think first

A CDN has four anycast sites and must remove one for maintenance. It serves short DNS requests and fifteen-minute customer file uploads. Design separate drain plans for both, including evidence from client networks and a rollback decision.

Take a few minutes to form an approach, then compare it with the worked explanation.

Reveal the worked explanation

Why DNS likes this arrangement

A DNS query is normally short and often independent of earlier queries. If it reaches a different healthy anycast resolver or authoritative server, that new location can answer without relying on the previous site's per-connection state. Anycast can reduce network distance and distribute some load. It also makes service observation tricky: a test from one network only confirms the node that its routing path currently selects.

An HTTP session is different. Suppose a client has a TCP connection to a CDN edge in city A. The server in A holds sequence-number state, congestion and flow-control state, TLS session state and possibly application routing decisions. The IP address in the client's socket remains unchanged. But after a route withdrawal or policy change, the client's packets begin arriving at city B. B usually has none of A's TCP connection state. Depending on the edge implementation, it may drop the packets or send a reset. The client experiences a broken connection despite unchanged DNS and destination IP.

Anycast does not necessarily break every long-lived connection. If the network keeps a flow on a stable path, or the operator has a design that preserves the relevant state, the connection can continue. The important risk is that routing changes can alter destination ownership in ways ordinary application load balancers do not control. That risk is greater for protocols whose sessions last much longer than the expected stability of the path.

The drain problem is not an on/off switch

Now plan maintenance for city A. We want new traffic to avoid A while not dropping active customer uploads. If we simply withdraw the site's announcement, remote networks may begin choosing city B. The withdrawal eventually changes the catchment, but it doesn't carry existing TCP state to B. Leaving A's processes running is not enough if the packets no longer arrive there.

A safer policy depends on the service. For stateless DNS, graceful site withdrawal is usually easier because requests are brief. For an HTTP upload, application-level resumability, smaller bounded transactions, an edge proxy terminating the connection, and controlled routing or draining can mitigate disruption. An architecture might keep a stable unicast address for sensitive sessions while using anycast for discovery or initial entry. Another might arrange stable edge termination and keep mutable user state in a separate backend. These are deliberate compromises, not universal recipes.

DNS-based steering is one alternative. It can return different site-specific addresses to clients, giving the operator more direct control over new connections. But cached answers can continue directing new connections to an old address, and established connections do not re-resolve DNS merely because a record changes. DNS steering and anycast can also be combined: DNS selects an anycast-enabled regional footprint, while internal routing selects an edge. Neither makes a stateless routing protocol a state replication system.

How I'd verify the incident

Record which anycast point of presence handled each request, not just the destination IP. Expose edge identifiers in safe response metadata or in internal telemetry. Measure traffic and resets by source autonomous system and network path. If an edge route changed at 14:07 and one population's established connections began resetting at the same moment, that is a more useful clue than a global average health score.

Test maintenance from multiple source networks, including clients with established sessions before withdrawal and new clients afterward. Watch whether packets continue to reach A, move to B or briefly have no route. A routing change can propagate at different speeds and affect different source networks differently. Also monitor connection age, active uploads and failed resumptions, so a successful withdrawal does not hide a user-visible incident.

The interview tradeoff is between coarse-grained network distribution and session ownership. Anycast excels where independent sites can answer a request with little location-specific state. When an application requires a long-lived stateful session, the design must assume that the network may change which site receives its packets. A stable service IP is not a promise of a stable server.

Go deeper

RFC 4786 describes anycast service operation, node placement, routing stability and protocol suitability. BGP policy selects paths, not a mathematically closest geographic point. Route withdrawal may shift a catchment and can strand existing stateful sessions. QUIC connection IDs can help a QUIC endpoint recognize a connection after a network path change, but anycast routing alone neither moves server state nor guarantees that packets reach the endpoint that has it. Anycast site-health signaling must account for partial failures and the service's actual ability to answer queries.

Check your reasoning

  • Explain one IP advertised from multiple routing locations and source-specific catchments.
  • Avoid promising geographically nearest placement or precise load balance.
  • Explain why moving TCP packets to a different anycast node can break session state.
  • Describe different maintenance drains for short stateless and long stateful requests.
  • Measure site ownership and client resets from multiple source networks.

Common mistake: An unchanged destination anycast IP means the same backend always owns a TCP session.

An anycast site must be removed while long-lived TCP sessions are active. Which plan addresses session ownership?

Read the answer explanation

Preserve stateful sessions through explicit edge design or application resumability — Routing can select another site for the same IP. Session ownership and recovery need explicit design.

Apply it elsewhere

An API gateway currently uses anycast for a WebSocket notification service. Evaluate whether it should retain the arrangement, introduce unicast session hosts or make reconnect-and-resume an explicit application contract.

Compare your approach

Describe shared prefix announcements, routing-based catchments, non-geographic path policy, TCP state ownership, site withdrawal, independent monitoring points and safe service-specific draining.

One more constraint

If two sites announce the same anycast IP, does the client's unchanged destination address guarantee that its established TCP connection will keep reaching the original site?

Quick revision

Anycast lets multiple sites advertise the same destination address, and routing selects the receiving site. This is useful for short, relatively stateless requests such as DNS. A route change can send packets from an existing TCP connection to a different site that lacks its state, even when the client's IP destination never changed. Maintenance must consider catchment shifts and long-lived sessions. Monitor actual edge ownership by source network, and use resumability or stable session termination when necessary.

References