Latency and throughput
TTFT, inter-token latency, tokens/sec, and why a fast average can still feel slow.
LLM Inference
Serve models with evidence, not intuition.
Move from a model endpoint to a production serving system: measure perceived latency, budget the KV cache, batch safely, choose precision, and plan capacity before the queue becomes the incident.
TTFT
<500ms
A useful interactive-chat target—not a universal SLA.
ITL
<30ms
A useful smooth-streaming target; verify by workload.
KV cache
<4% waste
PagedAttention paper result; actual headroom depends on model and scheduler.
Interactive visual playground
See the serving trade-offs move.
These are simulated teaching values—not live engine telemetry. Change the inputs to see why a serving configuration is always a latency, memory, and capacity decision.
Queue wait (ms)80
Prompt tokens800
Output tokens160
Simulated request waterfall1960 ms total
Queue
Prefill
Decode
Queue
80 ms
80 ms
Prefill
440 ms
440 ms
Decode
1440 ms
1440 ms
Free inference library
Understand each serving decision first
These public explainers and worksheets use measured results only where a source provides them. Hardware numbers, queue thresholds, and latency baselines are directional until you benchmark your own model, GPU, prompt mix, and concurrency.
KV cache and context
Size cache headroom, understand PagedAttention, prefix reuse, and long-context concurrency.
Batching and admission
Continuous batching, chunked prefill, queue guards, preemption, and SLO-aware routing.
Quantization trade-offs
FP16, FP8, INT8, and INT4 choices for weights and KV cache—with quality gates.
Serving stack
Choose vLLM, SGLang, TensorRT-LLM, llama.cpp, Triton, and routing/autoscaling layers.
Capacity planning
Turn SLOs, token mix, KV headroom, and utilization into replicas and runbook decisions.
KV-cache worksheet
How many sequences fit in the cache?
This calculator estimates cache occupancy only. Reserve additional memory for model weights, activations, CUDA/runtime overhead, and fragmentation.
128.0 KB
KV cache per token
512.0 MB
KV cache per sequence
20
Theoretical concurrent sequences
bytes/token = 2 × layers × KV heads × head dimension × bytes per element
bytes/sequence = bytes/token × context tokens
bytes/sequence = bytes/token × context tokens
Pro serving path
Apply the method to a real serving decision.
Pro turns the public concepts into a build sequence: create a latency baseline, size KV headroom, configure continuous batching, set admission limits, compare precision with an eval gate, and write a capacity runbook. Live Cohort adds an architecture review and deployment capstone.
Source discipline
The material draws from the LLM-inference guide and vLLM workshop repository, with upstream vLLM results explicitly marked as measured when used. Workshop configuration values are examples, not production defaults.