Continuous Batching and Admission Control for LLM Serving
[Definition] Continuous batching lets a serving engine add and remove requests between decode iterations instead of waiting for an entire fixed batch to finish. Admission control decides which requests may enter that shared work queue without violating latency and capacity objectives.
Why fixed batching wastes a generative GPU
Generation length varies. If a fixed batch waits for its longest request, completed requests hold slots while the long request continues decoding.
fixed batch: [short ✓] [medium ✓] [long ───────────────]
continuous batch: [new request enters] [completed request leaves] [long continues]Continuous batching improves utilization because the engine can fill free sequence slots as soon as a request completes. The vLLM paper reported up to 24× throughput versus Hugging Face Transformers and up to 3.5× versus TGI in its evaluated workloads; treat those as paper-specific measured results, not a guaranteed improvement for every deployment.
Prefill and decode compete for different resources
- ▸Prefill processes the input prompt, has substantial parallel work, and is often compute-heavy.
- ▸Decode produces one token per active sequence at a time, repeatedly reads KV cache, and is often memory-bandwidth-heavy.
Mixing them can improve device utilization, but an unbounded long prefill can delay all active decodes and damage ITL.
Chunked prefill
Chunked prefill limits how many prompt tokens one request consumes in a scheduler step. The remaining budget is available for decode requests and other prefills.
vLLM V1 documents chunked prefill as enabled by default: the scheduler prioritizes decodes, then fills remaining token budget with prompt work. The right max_num_batched_tokens is workload-dependent—smaller values can protect ITL; larger values can improve prefill throughput and TTFT under batch-oriented traffic.
The convoy effect
A single 100K-token prompt can monopolize prefill work while short chat requests wait. This is the convoy effect.
Mitigations:
- ▸cap prompt length at the product boundary;
- ▸chunk prefill instead of serving the full prompt in one step;
- ▸separate interactive and bulk queues;
- ▸reserve decode capacity for active streams;
- ▸route long-context/offline work to a different pool when the SLO differs.
Admission control is an SLO policy
Do not treat every request as equally urgent. A simple policy can use queue depth, waiting time, active decode count, cache occupancy, and request class.
if interactive_queue_wait > 250ms:
reject, shed, or defer batch requests
if gpu_cache_usage is high and preemptions rise:
lower max active sequences or add capacity
if prompt_tokens > long_context_limit:
route to long-context poolA polite overload response is better than accepting work that will time out after holding GPU memory for minutes.
vLLM signals to monitor
A useful serving dashboard includes:
| Signal | Why it matters |
|---|---|
num_requests_running | active decode/pre-fill pressure |
num_requests_waiting | queue growth and admission pressure |
gpu_cache_usage_perc | KV cache saturation |
| TTFT histogram | prefill and queue health |
| time per output token | perceived streaming smoothness |
| E2E latency histogram | end-to-end user experience |
| prefix-cache hit metrics | shared-prefix value |
| cumulative preemption count | scheduler overcommit signal |
vLLM V1 uses recompute as the default preemption mode. Rising preemption is a capacity/configuration warning: increase headroom, lower sequence/token limits, add parallelism, or scale replicas.
Workshop thresholds are examples, not defaults
The Ramu-DE/vLLM workshop config uses values such as a queue-size threshold of 5, running-request threshold of 10, one to four replicas, and max_num_seqs: 8 in an Inferentia exercise. Those are teaching configuration values, not a universal production policy. Calibrate against your model, accelerator, SLO, and burst pattern.
A practical rollout sequence
- ▸Start with a small concurrency cap and bounded queue.
- ▸Measure TTFT/ITL at steady load and bursts.
- ▸Enable continuous batching and chunked prefill.
- ▸Raise active sequences until P95/P99 or preemption crosses the SLO.
- ▸Add autoscaling on queue and running-request metrics, with scale-down stabilization.
- ▸Segment traffic classes before accepting large offline jobs into the chat pool.
Free concepts, Pro operation
The scheduler concepts are public. The Pro serving path turns them into an operating policy: define request classes, tune token budgets, set queue limits, interpret preemption, and rehearse overload handling. A Live Cohort capstone reviews the resulting serving SLO and autoscaling runbook.