LLM Serving Stack Glossary: Engines, Caches, Routing, and Metrics

[Definition] An LLM serving stack is the set of components that load a model, schedule token work, manage KV cache memory, route requests, scale replicas, and expose the latency and quality signals needed to operate the service.

The stack in one picture

text

Client / application
  ↓
API gateway + authentication + rate limits
  ↓
Router / queue / admission policy
  ↓
Serving engine (vLLM, SGLang, TensorRT-LLM, llama.cpp)
  ↓
GPU or accelerator + model weights + KV cache
  ↓
Metrics, traces, logs, autoscaler

Engines

Term	What it does	Good fit
vLLM	high-throughput serving with PagedAttention and continuous batching	general GPU serving baseline
SGLang	serving/runtime focused on structured generation and shared-prefix workloads	agentic and repeated-prefix applications
TensorRT-LLM	NVIDIA-optimized compilation and kernels	maximum NVIDIA-specific performance effort
llama.cpp	GGUF-oriented local/CPU/edge inference	laptop, edge, constrained environments
TGI	text-generation serving engine	Hugging Face ecosystem deployments
Ollama	local model lifecycle and developer convenience	local experimentation

Choose an engine based on hardware, model support, observability, deployment constraints, and workload shape—not a benchmark headline alone.

Memory and scheduling terms

Term	Meaning
prefill	processing the input prompt to create initial KV cache
decode	generating one output token per active sequence iteration
KV cache	stored key/value attention states for past tokens
PagedAttention	block-based KV-cache management that limits fragmentation
prefix caching	reusing cached KV blocks for identical prompt prefixes
continuous batching	add/remove requests between decode iterations
chunked prefill	split a long prompt across scheduler steps to protect decode latency
preemption	pause/recompute work when the scheduler overcommits cache capacity
`max_num_seqs`	cap on active sequences; an engine configuration, not a universal default
`max_num_batched_tokens`	token budget available to a scheduler iteration

Routing and scale terms

Term	Meaning
admission control	policy deciding whether a request joins, waits, sheds, or routes elsewhere
request queue	requests waiting for a replica/scheduler slot
replica	independent serving process with its own model and cache
tensor parallelism	split model tensors across accelerators to fit or accelerate the model
pipeline parallelism	place model layers across accelerators/stages
autoscaling	change replica count from demand/queue/metric signals
warm pool	replicas kept ready to avoid cold-start TTFT

Observability terms

Track these separately:

▸TTFT: time to first token;
▸TPOT / ITL: time per output token;
▸E2E latency: full request duration;
▸queue time: waiting before execution;
▸GPU/KV cache occupancy: cache saturation risk;
▸preemption count: scheduler overcommit signal;
▸prefix-cache hit rate: shared prompt reuse value;
▸tokens/sec: aggregate work rate, not a user-latency substitute;
▸cost per successful answer: ties serving efficiency to product value.

The Ramu-DE/vLLM workshop wires Prometheus metrics, KEDA scaling, and p50/p95/p99 tracing. Its queue thresholds and one-to-four replica settings are workshop examples; use them as a starting hypothesis only.

Platform layers

Layer	Role
Triton Inference Server	multi-model serving and metrics integration
NVIDIA NIM	packaged, managed inference microservices
Kubernetes/KEDA	replica scheduling and metric-driven scaling
Dynamo-style planner/router	SLO-aware routing and prefill/decode pool decisions
OpenTelemetry/Langfuse/CloudWatch	traces, token use, request outcomes, and latency analysis

Free concepts, Pro architecture

The glossary is public so architecture conversations use the same language. The Pro serving path selects an engine, maps the request flow, defines the metrics, writes admission rules, and ties autoscaling to an SLO. A Live Cohort capstone reviews the architecture and failure modes.