Most teams talk about latency as if it is one number.

Average latency. p50 latency. Total request time. Sometimes p95, if the team has already been burned once.

That is fine for a dashboard, but it is a weak way to understand how an LLM product actually feels. Users do not experience your average latency. They experience the first token. They experience whether the stream keeps moving or keeps pausing. They experience the one request that got stuck behind a queue. They experience the moment where the assistant looks alive, stops for six seconds, then continues as if nothing happened.

That is the part most dashboards hide.

Your LLM system may look fast at low traffic. The demo feels good. The staging numbers look clean. The p50 is fine. Even p95 may not scare anyone.

Then traffic grows, and the same system suddenly feels slow.

The model did not become worse. The prompts did not become worse. The GPU did not forget how to generate tokens. The tail changed because the load changed.

Tail latency is not a fixed property of the system. It is a property of the system under a specific load shape. The tail you measured at low QPS is not the tail your users feel at production QPS.

When there is very little traffic, most requests can be served directly. However, once there is more traffic, the requests have to queue up. The moment queuing enters the picture, p99 ceases to become an indicator for the slowest generation and starts becoming one for the queue.

Latency is not one thing in an LLM system#

When someone says “the LLM call is slow” that sentence is usually hiding four or five different problems.

There is time to first token. That is how long the user waits before anything appears. There is inter token latency. That is how smooth the stream feels after it starts. There is total generation time, which depends heavily on output length. There is queueing time before inference starts, where the request is accepted, but it is waiting for capacity.

All of these show up as latency, but they do not feel the same to the user.

A slow first token feels like the system is stuck. Bad inter token latency feels like the stream is choking. Long total generation time can be acceptable if the answer is long and the stream is moving. Queueing delay is worse because the user gets no signal. From their point of view, nothing is happening.

That is why the end timestamp alone is a bad metric for this product.

Two requests can both take 12 seconds.

In the first request, the first token arrives in 800 milliseconds and the answer streams smoothly for the next 11 seconds. That can feel fine.

In the second request, the first token arrives after 8 seconds, then the rest streams quickly. Same total time, very different product feel.

There is another problem. Users do not send one request. They have sessions.

If one request has a 1% chance of being painfully slow, that sounds small. But across a 50 request session, it becomes roughly a 40% chance that the session includes at least one bad experience.

That is why tail latency becomes trust latency.

A user can forgive one slow answer. But if the product keeps randomly freezing, they stop trusting it. They start thinking the system is unstable, even if your average latency is fine.

Average latency tells you what usually happened. The tail tells you what users remember.

The tail you measured is not the tail you will get#

The most dangerous latency number is the one you measured under the wrong load.

In staging, the system looks fine. Five requests per second, clean prompts, warm workers. little contention. The model starts quickly. The queue is mostly empty. p99 looks healthy.

Everyone feels good.

Then production traffic comes in. More users, longer prompts, mixed workloads, more RAG paths, more tool calls, more retries, more streaming sessions sitting open. Now the same system is doing the same work, but under a different load shape.

This is where the tail starts moving.

At low QPS, tail latency is mostly about the natural variance of the request. Some prompts are longer. Some outputs are longer. Some requests hit RAG. Some do not. Some need a tool call. Some do not.

At higher QPS, queueing becomes part of the latency. The request is not slow only because generation is slow. It is slow because it waited before generation even started.

Queueing does not get worse in a clean straight line. Near saturation, it gets ugly fast. A rough queueing intuition is that waiting pressure grows like utilization divided by one minus utilization.

At 50% utilization, that pressure is manageable. At 80%, it is already much worse. At 90%, the system is technically not full, but the wait can feel completely different. At 95%, small bursts can turn into a queue that takes time to drain.

The exact numbers will change with the serving stack, model size, GPU, request mix, batching policy and scheduler. But the shape is the same. The tail does not degrade politely. It bends, then it jumps.

This is why staging latency can lie without any metric being wrong.

The p99 at 5 QPS may be 3 seconds. At 25 QPS, it may be 5 seconds. At 45 QPS, it may become 14 seconds. At 55 QPS, it may become 40 seconds or start timing out.

Nothing magical happened between those points. The system got closer to saturation, and the queue became the product experience.

That is the part that hurts teams. They thought they measured the system. They only measured the system before the queue mattered.

The model latency did not explain the incident. The average did not explain it either. The tail moved because utilization moved.

This is also why “it worked in load test” is often a weak statement. Many load tests use clean prompts, fixed output sizes and steady traffic. Production is not steady. It has bursts. It has long conversations. It has users pasting large context. It has background jobs sharing capacity with interactive traffic. It has retries during provider issues.

That load shape is what creates the tail.

The useful question is not “what is the latency?” The useful question is “what is the latency when the system is close to the load shape it will actually see?”

If you do not answer that, you do not know your tail. You only know your demo.

The tail moves through the stack#

When p99 gets bad, people often ask which layer caused it.

Was it the model? Was it the GPU? Was it RAG? Was it tools? Was it guardrails?

The answer is usually annoying because it depends on the load level and the request shape.

At low load, the model generation path may dominate. A short prompt with a long answer spends most of its time decoding. A long prompt with a short answer spends a lot of time in prefill before the first token appears. If the system is not busy, the request enters quickly, runs and finishes. The tail mostly comes from input length, output length and the extra steps that request needed.

As load climbs, the dominant cost can move. Queueing before inference starts to matter. Now the GPU being busy is not just a utilization graph. It becomes user visible waiting time. The request has not even started producing tokens, but the user is already paying latency.

Context length makes this worse. Long context is not just “more tokens.” It increases prefill cost. It increases memory pressure through the KV cache. It can lower the effective throughput of the system, which means queueing can start earlier than expected.

This is one reason RAG systems get surprised by latency. A RAG pipeline may retrieve five chunks in testing, then ten chunks in production because someone increased top k to improve answer quality. Maybe each chunk is larger now because the team changed chunking. Maybe conversation history is also included. The prompt got bigger, first token got slower and capacity dropped.

The change looked like a quality improvement. It was also a latency change.

Tool calls add another kind of tail because they are serial steps. The model decides to call a tool, the app calls the tool, the tool waits on another service, the response comes back, then the model continues. One slow dependency in that chain can dominate the request, and it may not show up when you only look at model latency.

Guardrails and policy checks add their own tax. Sometimes it is small and predictable. Sometimes it means another model call before or after generation. The cost may look acceptable at p50 and still hurt at p99 when combined with RAG and tools.

This is why “LLM latency” is a messy phrase. The slow part may be before the model, inside the model, after the model or because of the queue around the model.

The LLM specific part is generation length variance.

In many backend services, requests are not equal but they are often close enough that the queue is easier to reason about. In LLM systems, one request may generate 40 tokens and another may generate 2000. One request may have a 700 token prompt, another may have 40000 tokens. One may finish quickly. Another may hold capacity for a long time.

Continuous batching helps, but it does not make variance disappear. Long sequences still occupy KV cache. They still consume decode steps. They still affect scheduling. When the system is near saturation, a few long requests can change the tail for other requests behind them.

That is why LLM queueing feels different from a normal API queue.

The cost of a request is not known cleanly at admission time. Output length is not fully known. Tool paths are not always known. RAG size may vary. User behavior varies. The scheduler is trying to keep hardware busy while the product is trying to keep users feeling like the system is responsive.

Those goals are related but they are not the same.

High GPU utilization looks good for cost. High GPU utilization near the edge can destroy the tail.

That is the tradeoff teams have to see clearly.

Measure the tail your users will actually hit#

You cannot fix tail latency by staring at one latency number harder.

You need to measure the path.

Start with time to first token. If first token is slow, the product feels slow before the answer even exists. Split that into upstream time, queue time, prefill time and everything else that happens before the first token.

Then measure inter token latency. A response that starts fast but streams unevenly still feels bad. The user sees the system hesitate. It breaks flow.

Then measure total time but do not compare total time across outputs with very different lengths as if they mean the same thing. A 100 token answer and a 2000 token answer should not sit in the same bucket without context.

Measure queue time directly. If queue time is not visible, you will blame the model for latency that was created before the model ran.

Measure under real load. Use bursts. Use mixed prompt sizes. Use long conversations. Use RAG heavy requests. Use tool heavy requests. Use the same retry behavior production uses. If background jobs share capacity, include them too.

This is where many teams cheat without meaning to. They load test with clean synthetic prompts and then wonder why production behaves differently. The test was easier than production. The tail was never really tested.

The SLO also has to match the product.

For a chat product, time to first token may matter more than total generation time. For an agent workflow, step level latency and timeout behavior may matter more than single response latency. For a coding assistant, stream smoothness matters because the user is watching code appear.

There is no universal latency SLO that works for every LLM product. The product shape decides what latency means.

That is the uncomfortable part. You cannot copy a number from another team and call it reliability.

You have to decide what the user is waiting for.

Then measure that thing under the load where the system actually runs.

The tail is the product#

LLM latency problems usually start with a simple sentence: “the model is slow.”

Sometimes that is true.

A bigger model may be slower. A long context may be slower. A long answer may be slower.

But in production, the more useful question is where the tail is coming from and what load shape created it.

At low QPS, your system can look healthy because requests rarely wait. At scale, the queue becomes visible. Context bloat makes each request heavier. RAG and tools add serial hops. Guardrails add tax. Long generations hold capacity. The tail starts to represent the whole system, not only the model.

That is why the tail you measured once is not enough.

Measure time to first token, stream smoothness, queue time, easure by route, token length, tool path and load level. Load test with traffic that looks like production, not traffic that makes the chart look good.

Users do not experience your average. They experience the request that got stuck.

And if that happens often enough, they will not describe your system as “p99 degraded under load.”

They will just say it is slow.