SPEC: Latency
Definition
[Definition] Latency is the time elapsed between submitting a prompt to an LLM and receiving the output. It encompasses network transmission, server-side queuing, model computation (prefill + decode), and response delivery. Latency is a critical quality-of-service metric for LLM applications.
Latency Components
Total Latency = Network (client→server)
+ Queue wait time
+ Prefill time (process input tokens)
+ Decode time (generate output tokens × N)
+ Network (server→client)
Key Latency Metrics
Time to First Token (TTFT)
- ▸Time from sending request to receiving the first output token
- ▸Dominated by: network RTT + queue + prefill computation
- ▸Critical for perceived responsiveness in chat UIs
- ▸Streaming enables users to see output while generation continues
Time per Output Token (TPOT) / Tokens per Second (TPS)
- ▸How fast the model generates each subsequent token
- ▸TPOT = milliseconds per token; TPS = tokens per second (inverse)
- ▸Dominated by: memory bandwidth (reading KV cache + weights)
- ▸Typical: 20–150 TPS for hosted APIs
End-to-End Latency
- ▸Total time from request to complete response
- ▸= TTFT + (output_tokens × TPOT)
- ▸Relevant for batch jobs and non-streaming use cases
Time Between Tokens (TBT)
- ▸Consistency of generation speed across tokens
- ▸High variance → choppy streaming experience
Typical Latency Ranges (2024)
| Model / Service | TTFT | TPS |
|---|---|---|
| GPT-4o | ~0.5–1s | 40–80 |
| Claude 3.5 Sonnet | ~0.5–1.5s | 50–100 |
| GPT-3.5 Turbo | ~0.3–0.5s | 80–150 |
| Local LLaMA 3 8B (GPU) | ~0.1–0.3s | 50–200 |
| Local LLaMA 3 70B (GPU) | ~0.3–0.8s | 15–50 |
Factors Affecting Latency
Input Side
| Factor | Effect on Latency |
|---|---|
| Prompt length (tokens) | ↑ Prefill time linearly |
| Context window usage | ↑ Prefill time |
| KV cache miss (no caching) | ↑ TTFT significantly |
Model Side
| Factor | Effect on Latency |
|---|---|
| Model size (parameters) | ↑ Both TTFT and TPOT |
| Number of layers | ↑ TPOT |
| Attention head count | ↑ Memory bandwidth requirements |
| Quantization level | ↓ TPOT significantly |
Output Side
| Factor | Effect on Latency |
|---|---|
| Output length (tokens) | ↑ Total latency linearly |
| max_tokens setting | Doesn't affect speed, only when to stop |
Infrastructure Side
| Factor | Effect on Latency |
|---|---|
| GPU memory bandwidth | ↓ TPOT when higher |
| GPU count (tensor parallel) | ↓ TPOT |
| Server load / queue depth | ↑ TTFT when high |
| Geographic region | ↑ Network RTT when far |
| Prompt caching hit | ↓ TTFT significantly |
Latency vs. Throughput Trade-off
| Optimization | Latency Effect | Throughput Effect |
|---|---|---|
| Small batch size | ↓ (good for latency) | ↓ (bad for throughput) |
| Large batch size | ↑ (bad for latency) | ↑ (good for throughput) |
| Speculative decoding | ↓ Latency | ≈ Same or slightly better |
| Quantization (INT8/INT4) | ↓ Latency | ↑ Throughput |
Latency Reduction Techniques
Speculative Decoding
- ▸A small "draft" model generates K tokens quickly
- ▸The large "target" model verifies all K tokens in one forward pass
- ▸Accepted tokens are kept; rejected tokens fall back to the target's output
- ▸Net effect: 2–3× speedup at same quality
Prompt Caching
- ▸Cache the KV states for repeated system prompts / documents
- ▸Cache hit → prefill cost drops by ~90%
- ▸Supported by: Claude (Anthropic), GPT-4o (OpenAI prompt caching)
Quantization
- ▸INT8 or INT4 weights → smaller memory footprint → faster memory reads
- ▸Modest quality loss, significant speed gain
Flash Attention
- ▸Reorders attention computation to be I/O efficient
- ▸Reduces memory bandwidth requirements for attention
- ▸2–4× faster attention computation
Streaming
[Key Insight] Not faster — just makes latency feel lower. First tokens appear immediately; users read while rest generates.
Smaller Models
- ▸Use the smallest model that meets quality requirements
- ▸GPT-3.5 vs. GPT-4: ~5× cost, ~3× faster
Latency SLAs (Service Level Agreements)
Common real-world targets by use case:
| Use Case | TTFT Target | TPOT Target |
|---|---|---|
| Interactive chat | < 500ms | < 30ms |
| Autocomplete / copilot | < 200ms | < 20ms |
| Async document processing | < 5s | Any |
| Background batch jobs | No TTFT target | Throughput-focused |
Measuring and Monitoring Latency
- ▸LangSmith, Helicone, Braintrust — LLM observability platforms
- ▸Custom logging: timestamp at request, first chunk, last chunk
- ▸p50 / p95 / p99 percentiles — don't rely on averages
- ▸Test under realistic load (latency degrades under high concurrency)
Related Concepts
- ▸Inference, Token, Context Window, Streaming, KV Cache, Throughput, Quantization, Prompt Caching