LLM Inference Capacity Planning: From Token Mix to Replica Count

[Definition] LLM capacity planning translates a workload—prompt length, output length, concurrency, burst pattern, and latency SLO—into model-memory, KV-cache, throughput, and replica requirements. It is a benchmark exercise, not parameter counting.

Start with the workload, not GPU FLOPS

Collect a representative distribution:

requests per second: average / peak / burst duration
prompt tokens:       P50 / P95 / P99
output tokens:       P50 / P95 / P99
request class:       interactive, batch, long-context, background
SLO:                 TTFT, ITL, E2E, error / shed rate

A 512-token benchmark does not size a product where P95 prompts are 20K tokens and users stream 1K-token answers.

Memory budget first

A serving replica needs room for:

accelerator memory
− model weights
− runtime / CUDA / communication overhead
− activation and workspace reserve
= KV-cache headroom

For a Llama-3.1-8B-like architecture with 32 layers, 8 KV heads, 128 head dimension, and BF16 KV values:

KV bytes/token = 2 × 32 × 8 × 128 × 2
               = 131,072 bytes ≈ 128 KB/token

KV bytes/4K-token sequence ≈ 512 MB

If only 10GB remains after weights and overhead, the theoretical cache limit is roughly 20 such 4K sequences. The practical limit is lower because scheduler fragmentation, output growth, and latency SLO headroom matter. Use the KV-cache worksheet with actual model dimensions.

Example 1: interactive 8B assistant

Assumptions — planning example, not benchmark result

Plan:

1. ▸validate model plus runtime fit on the chosen accelerator;
2. ▸reserve KV headroom for the desired active sequences;
3. ▸benchmark 1, 4, 8, 12, and 16 active requests;
4. ▸choose the highest concurrency that keeps P95 TTFT/ITL within SLO;
5. ▸set autoscale triggers before waiting queue exceeds the interactive wait budget.

The Ramu-DE/vLLM workshop's target of under two seconds for a 512-token response on Inferentia is a workshop target, not a measured capacity promise for another model or workload.

Example 2: shared system prompt with prefix caching

A support assistant has a 2K-token system prompt and policy pack shared by most requests. Without prefix caching, every request pays the prefill cost and stores duplicate cache blocks. With automatic prefix caching enabled, identical prefixes can reuse KV blocks.

Capacity implication: benchmark two traffic mixes—cold unique prompts and warm shared prefixes. Report prefix-cache hit rate alongside TTFT. Do not assume a fixed speedup; the benefit is determined by exact-prefix reuse and eviction pressure.

Example 3: long-context document analysis

A document-analysis workload accepts 32K-token prompts and produces short answers. Here prompt prefill and KV occupancy dominate. Put it in a separate queue or pool from chat traffic:

• ▸apply a higher TTFT allowance;
• ▸use chunked prefill;
• ▸limit simultaneous long contexts;
• ▸benchmark answer quality under any context truncation policy;
• ▸prevent the long-context queue from consuming interactive decode capacity.

Replicas from measured sustainable rate

After benchmarking, calculate with headroom:

required replicas = peak request rate / sustainable request rate per replica
then add failure and burst headroom

Use sustainable rate at the chosen SLO, not maximum tokens/sec observed in a saturated benchmark. A simple starting policy might target 60–75% steady-state utilization so bursts, model variance, and a failed replica do not immediately create an unbounded queue.

Autoscaling signals

Use more than utilization:

The vLLM workshop's KEDA settings—queue threshold 5, running requests 10, one to four replicas—are reasonable example inputs for a lab, not production defaults.

Cost and quality belong in the plan

Capacity is not complete until it includes:

• ▸precision choice and eval result;
• ▸model/engine version;
• ▸accelerator and reservation/on-demand rate;
• ▸target utilization and replica count;
• ▸tokens in/out per successful user task;
• ▸fallback, overload, and rollback policy.

A lower-cost configuration that causes retries or unsupported answers can cost more per successful outcome.

Free method, Pro runbook

This method is public. The Pro serving path applies it to a real workload: collect traces, size cache headroom, benchmark concurrency, separate traffic classes, select autoscale signals, and write a capacity/runbook decision record. A Live Cohort capstone reviews the plan under burst and failure scenarios.