What the limit actually limits
Under the Linux cgroup v2 CPU controller, cpu.max represents a quota and a period. For example, 200000 100000 means the cgroup is entitled to up to 200,000 microseconds of aggregate CPU runtime during a 100,000-microsecond period. That is two cores worth of CPU time, not a promise that precisely two threads will always run.
Imagine a request burst wakes eight runnable workers. With eight available physical CPUs, those threads can collectively burn 200 milliseconds of runtime in about 25 milliseconds of elapsed time. The quota for the current 100 ms period is now exhausted. Under ordinary bandwidth control, the runnable group cannot use more quota until the next period. Even if the rest of the machine has idle cores, those threads can wait.
A request arriving just after quota exhaustion might sit unable to execute Java code, finish a TLS handshake, or emit its response. The same request arriving early in the next period might be fast. Your p99 reflects the unlucky placement in time.
The example assumes a normal 100 ms period and enough parallel runnable work to consume two-core quota rapidly. Configured period length, quota, ancestor cgroups, and burst settings change the precise behavior. Do not claim every throttled period necessarily adds 75 ms to every request.
Find out whether throttling is actually the culprit
First correlate latency spikes with the rate of change in the cgroup's cpu.stat, especially nr_periods, nr_throttled, and throttled_usec. These counters are cumulative. An absolute total since startup is not actionable. A rising throttled-period fraction and substantial throttled time during the bad windows are much stronger evidence.
Check cpu.max in the affected cgroup rather than trusting deployment YAML alone. A Kubernetes CPU limit may be translated into cgroup quota, but admission policy, runtime configuration, and parent cgroups can affect the effective budget. Confirm the kernel's cgroup version and inspect the container and its ancestors if the observed quota seems inconsistent.
Now compare runnable thread pressure with actual CPU usage. CPU usage around 0.8 cores does not tell you whether eight workers are briefly fighting for four available CPUs or two cores of quota. Track run-queue delay, application thread-state samples, request arrival bursts, GC activity, and the concurrency of work that wakes simultaneously. CPU quota and JVM garbage collection can interact because collector threads also need CPU.
I would run a controlled test at the same offered traffic, once with the current CPU limit and once with a higher or absent limit, while retaining an appropriate CPU request and observing node contention. If tail latency improves and the cpu.stat throttling delta collapses, that supports the diagnosis. It does not make unlimited CPU safe for every tenant sharing that node.
Why simply adding replicas can disappoint
Adding replicas may spread requests but only if the balancer and traffic patterns actually spread those bursts. A single request that uses a pool of eight CPU-heavy workers can still exhaust a two-core pod's quota. A synchronized refresh or scheduled job might trigger all pods at once. Conversely, removing the limit may improve latency for this workload while making unrelated pods miss their performance budgets through noisy-neighbor contention.
I would determine the workload's burst envelope before choosing a setting. A request with 20 ms of CPU work spread over four parallel phases has a different scheduling profile than a long computation that needs two whole cores continuously. Thread counts, executor limits, and the amount of work performed on the critical path matter. Restricting concurrency can sometimes make p99 better by avoiding bursts and queue growth, but it can also lower throughput, so measure both.
Consider an autoscaler, too. If it scales from minute-level CPU utilization while the real problem occurs in 25 ms bursts, it can miss the signal. Scale decisions may need request concurrency, queue delay, or a latency-relevant saturation metric, with safeguards against oscillation. Longer CPU windows do not erase a short scheduling cliff.
What I would change
For a latency-sensitive dedicated API, I would test a CPU request that reflects actual resource needs and reconsider an unnecessarily tight CPU limit, subject to cluster policy and fairness requirements. In an environment with strict quotas, I might increase the quota, reduce per-pod burst parallelism, or shift batch work to separate pools. Each option addresses a different part of the problem.
The interview answer should end with evidence. Show that throttling and p99 are correlated, establish a controlled causal improvement, and check the blast radius. The lesson is not that CPU limits are bad. It is that average utilization is the wrong measurement of a scheduler enforcing a budget over much shorter intervals.
