The unit “one request” has become meaningless for capacity. A video input may need fetch or upload, decoding, frame sampling, resize and crop, vision encoding, and insertion of visual representations into the model context. Exact tokenization and patch counts depend on the model and processor. A 20 second clip at two sampled frames per second and a clip at 16 frames per second have very different costs even with the same text prompt. vLLM's multimodal input documentation describes video and image paths, per-prompt limits, and model-specific pruning behavior. Its multimodal processor benchmark separates processor stages through encoder execution. I would measure this path for the actual model rather than multiply video seconds by a universal “tokens per frame” number.

Trace one request from upload to first token. Record bytes, frames received and selected, resolutions, CPU decode time, encoder queue and compute, visual token or embedding count, language prefill, scheduling wait, and KV occupancy. Break p99 down by modality and tenant. If the model server reports only text tokens, it can look healthy while a preprocessing pool is saturated or vision embeddings have made prefill larger. An average across text and video is especially misleading when a small expensive slice owns the tail.

Admission should have budgets for bytes, frames, resolution or derived vision work, total model context, deadline, and concurrent encoder occupancy. Apply cheap bounds before downloading or decoding unbounded media. The scheduler needs a cost estimate refined after preprocessing, then a way to reject or queue work without occupying scarce GPU memory while it waits. Separate interactive requests from long video analysis if their latency contracts differ. Keep image fetch and video decoding inside the authorized data and network boundary too.

Product can keep long clips, but it has to choose a representation. Sample frames or segments guided by the task, then evaluate whether the summary still catches short, decisive events. A violent scene, a one-frame error code, and a speaker change fail differently under uniform pruning. For tasks requiring frame-accurate evidence, process windows asynchronously with timestamps and cite the relevant window. If someone says “just reduce to one frame per second,” show the recall curve by event duration and the actual latency saved. A serving optimization that drops the only evidence is a product regression.