Topic reading guide
LLM infrastructure
Request counts hide much of the work in an LLM platform. Build a model of serving capacity, admission and tenant isolation before choosing routing or retry policies.
Start with serving mechanics
LLM Inference Is a Scheduler Problem
See how prefill, decode, continuous batching and KV memory determine what an inference engine can safely run.
Essay
Measure load and latency
Your LLM Dashboard Is Lying About Load
Replace request-count intuition with signals that reveal token, memory, queue and scheduler pressure.
Essay
Your LLM System Isnt Slow. Your Tail Is Slow.
Separate time to first token, stream pauses and queue delay to find where users actually wait.
Essay
Control admission and capacity
Your LLM Rate Limiter Is Counting the Wrong Thing
Reserve for uncertain token work before it reaches the backend, then reconcile actual use.
Essay
One Tenant Can Break Your LLM Platform Without Sending Many Requests
Design fair queues, KV budgets and isolation for workloads whose request counts hide their resource use.
Essay
Route and recover deliberately
Your LLM Router Can Make Cheap Models Expensive
Choose routes across a session using cache locality, quality, provider health and switching cost, not token price alone.
Essay
Retries Make LLM Systems Less Reliable
Bound retries and fallback so provider failures do not amplify load, cost or behavior changes.
Essay
Keep reading
Continue with a guided sequence of free production engineering essays.
Find your next reading path