Skip to content

The database is healthy. Why can't the pod reach it?

The Route Exists. Does the Request Arrive? · Part 1 of 6 · 5 min read

Read the worked explanation ↓

Interview question. An order service runs in a Kubernetes pod in a VPC in Mumbai. Its database sits behind a private endpoint in another region. The application times out, yet the database is accepting connections locally and both infrastructure dashboards are green. Trace the packet, explain which network controls can prevent it from arriving, and diagnose the failure without immediately adding a NAT gateway.

Start with the destination address, not the ping command. The application asks a resolver for orders-db.internal. That name might resolve to a private database interface, a proxy, or an endpoint visible only to the VPC's DNS configuration. A correct hostname resolution does not prove the route exists, and a failed lookup is a different problem from a successfully resolved address that cannot be reached.

Suppose the source VPC uses 10.24.0.0/16 and the destination uses 10.72.0.0/16. These are illustrative private address ranges. The pod opens a connection to 10.72.18.40:5432. Whether that source packet retains the pod's IP address depends on the Kubernetes network implementation, its CNI and the path. Some networking setups route pod addresses directly. Others translate pod addresses to node addresses for particular destinations. Never assume all pod-to-VPC traffic is masqueraded.

A pod's database packet moves through a VPC route table, a private inter-region connection and the destination's controls, with a return route required.
A pod's database packet moves through a VPC route table, a private inter-region connection and the destination's controls, with a return route required.
Think first

Two Kubernetes clusters live in separate regions. Cluster A uses `10.24.0.0/16`, cluster B uses `10.72.0.0/16`, and only one namespace cannot reach port 5432. Propose a diagnosis that localizes DNS, source translation, routing, return traffic, firewall and database TLS failures without weakening production access controls.

Take a few minutes to form an approach, then compare it with the worked explanation.

Reveal the worked explanation

Follow the forwarding decision

The packet leaves the pod network, passes through the node's forwarding path and reaches the VPC's virtual routing machinery. The source subnet route table must have a destination route matching 10.72.0.0/16, pointing to an appropriate private connection. That might be a VPC peering connection, a transit gateway attachment or another approved network design. A default route to an internet gateway does not automatically make a private database address globally reachable.

Now follow the network hop into the other region. A VPC peering connection carries private traffic between two supported VPCs, including across regions, but each side needs the correct route entries. Peering is not transitive. If VPC A peers with B, and B peers with C, A cannot assume it can reach C through B. A transit gateway can simplify routing across many attached VPCs, but it brings its own attachment, route-table and peering configuration. Those are decisions, not invisible network magic.

At the destination, inspect ingress policies for the actual observed source, after any permitted address translation. The database or proxy may have a security group admitting one node CIDR but not a new pod CIDR. A network ACL can reject packets in one direction because its rules are stateless. Security groups on AWS are stateful, but being allowed through one control does not override a deny in another. If the SYN reaches the destination and the SYN-ACK does not return, investigate reverse routing and stateless response-port rules instead of changing application connection pooling.

This is why a one-way architecture diagram is insufficient. The return path also needs a route to the source address the destination actually sees. If an SNAT gateway changes the source, the reply follows the translated source. If the design preserves the pod source, the return routing must understand that prefix. Asymmetric routing through stateful middleboxes can break flows even if forward and return routes each appear valid in isolation.

The address collision nobody spotted

The interviewer now tells you that an acquired service already uses 10.24.0.0/16 in its destination VPC. An overlapping private CIDR does not merely create a cosmetic naming conflict. Ordinary direct VPC peering cannot be established between VPCs with overlapping CIDRs on AWS. A router may consider 10.24.4.8 local when you meant a remote endpoint. A workaround needs an explicit design, such as renumbering, an intermediary service endpoint, or carefully controlled translation, not a broader 0.0.0.0/0 rule.

That changes the design decision. For a single database service, exposing a narrow, authenticated private service endpoint may be safer than granting two entire networks broad connectivity. A database proxy or managed private endpoint can give consumers a routable interface without requiring every client to know the underlying database CIDR. But confirm the product's cross-region support and initiation direction, rather than treating all private endpoint products as interchangeable.

How I would investigate

I would resolve the database name from the failing pod, record A and AAAA answers, then inspect the resulting source and destination addresses. Next, compare a failing pod with a working one using the exact destination port. If failures are confined to one namespace, inspect the NetworkPolicies selecting those pods, including DNS and database egress, and confirm the network plugin actually enforces them. Check the CNI egress behavior, source subnet route table, private-connection attachment state, destination subnet routing, security policies and return path. VPC flow logs and packet captures, where supported, can distinguish drops, missing responses and successful handshakes followed by TLS or application failures. Flow logs are evidence with collection limits, not a guarantee of recording every packet.

Finally, validate the application's authentication and TLS endpoint name. An open TCP connection does not prove a usable database session. The key interview skill is drawing a packet-level path with a verifiable route and security decision at every boundary. Once those facts are known, adding a gateway, relaxing a firewall or changing a timeout stops being guesswork.

Go deeper

For AWS VPC peering, both VPCs require route-table entries for the desired address ranges, and overlapping VPC CIDRs prevent creating the peering connection. Transitive forwarding through a third peered VPC is unsupported. Transit gateway attachments use separate association and propagation controls, and cross-region gateway peering requires explicit routes. Kubernetes pod address translation depends on its network plugin and routing configuration. If private endpoints are used instead, verify service-direction support and endpoint DNS behavior, which are not the same thing as general-purpose routed VPC connectivity.

Check your reasoning

  • Resolve the hostname and distinguish DNS from actual route reachability.
  • Show the real pod or node source after CNI translation.
  • Trace both VPC subnet routes, peering or transit attachments, and return traffic.
  • Check security group state, stateless network ACL rules and TCP/TLS separately.
  • Explain CIDR overlap and a least-privilege service access alternative.

Common mistake: Private DNS resolution proves the pod has end-to-end private connectivity to the database.

A pod resolves the database hostname but its connection times out. Which investigation should come first?

Read the answer explanation

Trace the actual source IP, forward and return routes and destination controls — DNS answers only identify a destination. Address translation, routes, firewall policy and the return path determine reachability.

Apply it elsewhere

A newly acquired business uses the same private CIDR range as your application VPC. Design service-to-service access that avoids an unrestricted full-network bridge and still supports audit and failover.

Compare your approach

Trace the actual source after CNI decisions, destination DNS mapping, longest-prefix route choices, per-boundary security, return path, overlap restrictions, and how to prove which stage failed.

One more constraint

If a pod resolves a private database address successfully, does that prove its packets have a working bidirectional private route to the destination?

Quick revision

Start with the resolved destination and the actual source address after any CNI translation. Trace the outbound subnet route, private connection, destination ingress control, and return route. AWS VPC peering is not transitive and overlapping CIDRs prevent direct peering. Security groups and network ACLs have different state behavior. DNS success does not imply routing success, and TCP success does not imply database TLS success. Diagnose each boundary before changing architecture or timeouts.

References