How the data ended up on the wrong node
Many Linux allocations follow a local-preferred policy: physical pages are generally assigned when first touched, subject to memory policy and allowed-node constraints. The thread that initializes a large cache can influence which NUMA node owns its pages. Suppose one startup thread builds a 90 GB memory-mapped index on node 0, then the query workers are scheduled across nodes 0 through 3. A large portion of their random lookups can travel across socket boundaries.
Another possibility is that the process is pinned to cores on one node while the requested memory cannot fit there. Pages may be allocated remotely or allocation can fail under stricter binding policies. The correct outcome depends on the memory policy, node capacity, cpuset, page placement, and kernel behavior. Do not assume the scheduler automatically fixes everything or that every remote allocation is a bug.
Random access is especially sensitive. A database performing many dependent pointer dereferences might stall on each memory access before it knows where the next pointer leads. A sequential scan can sometimes prefetch effectively and achieve high bandwidth, even if some pages are remote. So I would look for a workload-specific signal, not simply say that NUMA always makes memory slow.
Measurements that distinguish NUMA from a slow database
Start with hardware topology: lscpu, numactl --hardware, socket layout, and which NUMA nodes a database process is allowed to use. Inspect memory placement via /proc/<pid>/numa_maps, or controlled tooling that presents the per-node distribution. Check cpuset boundaries and whether container orchestration has actually reserved particular CPUs and NUMA nodes.
Then measure the running workers. Are threads migrating across sockets? What share of memory references is local versus remote according to available performance counters? Are last-level-cache misses and memory access stalls elevated? perf and platform-specific NUMA counters may help, but exact event names vary by processor. Compare query classes, not just overall QPS. An index lookup dominated by random memory access can show a very different response from a full-table scan.
I also want to rule out false leads: lower processor frequency on the new SKU, SMT differences, a new kernel, changed page sizes, memory bandwidth saturation, or another tenant using the interconnect. A 30% latency regression after migration is an observation, not automatically proof of remote memory.
A controlled experiment beats a NUMA slogan
Run the same representative read-only workload on a constrained CPU and memory placement within one socket, using enough local capacity. Then test spreading workers across sockets while leaving memory placement fixed, and finally a policy that places workers and their allocated pages together. Keep QPS, warm-up, page-cache state, and memory footprint comparable.
Binding is not free. If node 0 holds only 64 GB, you cannot bind an honest 90 GB active working set to that node and expect it to fit. A strict memory bind may fail allocations rather than spilling anywhere. An interleave policy can distribute memory across nodes and smooth bandwidth demand, but that deliberately sacrifices some locality. For a workload dominated by bandwidth rather than access latency, this may still be a better outcome.
Another design is to split the database into independent partitions or processes whose workers and hot pages live close together. Each partition owns a subset of keys or files. A request coordinator routes queries to the right owner and combines results when needed. That introduces cross-partition query costs and operational complexity. It is not a free improvement if most requests need every partition.
Database cache policy complicates the picture
Some databases use application-managed buffer pools. Others rely heavily on the operating system page cache. Anonymous buffer-pool pages, mapped files and file-cache pages may respond differently to memory policies and migration. Huge pages, transparent huge pages, page migration, and automatic NUMA balancing can change the practical result. Do not assume one numactl switch relocates every page an already-running process touched.
Think about warm-up. A process that builds a cache with one initializer thread and later launches workers across four sockets can recreate poor placement on every deployment. Startup placement, allocator arenas, thread affinity and the ownership model should be designed together. Otherwise a benchmark that starts workers differently from production can give a misleading answer.
The Principal-level decision is whether this should be one enormous shared-memory process at all. Sometimes a two-socket machine with fewer, faster local accesses wins. Sometimes four sockets deliver the needed capacity if data and computation are placed intentionally. Capacity planning needs memory topology, not only the sum printed by free.
