LLM Inference Latency: TTFT, Inter-Token Time, and Throughput

[Definition] LLM latency is not one number. Time to first token (TTFT) measures how long a user waits before streaming begins; inter-token latency (ITL) measures how smooth generation feels; throughput measures how much work the serving system completes over time.

Split the request into prefill and decode

request accepted → queue → prefill prompt → first token → decode token 2…N → final token

A fast total response can still feel slow when TTFT is high. Conversely, a low TTFT with 100ms ITL produces a visibly sluggish stream.

Directional hardware baselines

The LLM-inference guide reports this directional, third-party benchmark example for a 7B-class model with a 350-token prompt and 150-token output. It is not an SLA and should not be copied into a production contract.

A 70B-class model can require 50–100ms or more per generated token depending on precision, parallelism, prompt length, and batch pressure.

[Key Insight] Use TTFT under 500ms and ITL under 30ms as useful interactive-chat targets, not universal values. Measure on your model, GPU, tokenizer, prompt mix, and concurrency profile.

What moves each part of the curve

TTFT

TTFT rises with queueing, prompt length, cold model loads, prefix-cache misses, and large prefill batches. Chunked prefill can prevent one long prompt from blocking short interactive requests.

ITL

Decode is commonly bandwidth-bound. ITL rises when the KV cache consumes headroom, too many sequences decode together, preemption occurs, or tensor-parallel communication dominates.

E2E

A useful approximation is:

E2E ≈ queue + TTFT + (output_tokens - 1) × ITL

For a 300-token answer, an extra 10ms of ITL adds about three seconds after the first token.

Build a latency baseline before tuning

Record at least these dimensions with every benchmark:

• ▸model and precision;
• ▸GPU/accelerator and parallelism setting;
• ▸prompt-token and output-token distributions, not one fixed prompt;
• ▸concurrent requests and queue policy;
• ▸prefix-cache hit rate and cache occupancy;
• ▸TTFT P50/P95/P99, ITL P50/P95/P99, E2E P50/P95/P99;
• ▸errors, cancellations, and preemption count.

Use separate tests for a single request, steady concurrency, burst traffic, long prompts, and long outputs. A system that passes the single-request demo may fail the burst test through queue growth.

Practical interventions

Continue the inference path

Read KV Cache before increasing concurrency, then Continuous Batching and Admission Control to protect the tail. The LLM Inference area contains the KV sizing worksheet and the Pro serving path.