LLM Serving Quantization: Memory Savings Need a Quality Gate

[Definition] Serving quantization stores or computes model weights and sometimes KV cache values at lower precision to reduce memory traffic and increase feasible concurrency. It is an engineering trade-off, not a free speed multiplier.

First: know what is being quantized

Target	Why it matters	Typical consequence
weights	determines whether a model fits in accelerator memory	lower memory and bandwidth demand
KV cache	determines active-context concurrency	more sequences or longer contexts fit
activations	affects kernel and numerical behaviour	hardware/framework dependent
optimizer state	training concern, not serving	usually irrelevant to inference

Do not apply vector-database quantization figures to LLM serving. A 4× memory reduction for a search index says nothing about an LLM's answer quality or decode speed.

Precision choices

Format	Approx. bytes/value	Best starting use	Main caution
BF16 / FP16	2	quality baseline	largest weight and KV footprint
FP8	1	supported Hopper/Blackwell serving stack	kernel and model support required
INT8	1	memory-constrained weight/KV scenarios	evaluate task-quality impact
INT4	0.5	fitting larger models into limited memory	higher quality and kernel risk

The byte ratio is not the delivered throughput ratio. Decode throughput can be limited by memory bandwidth, kernel efficiency, parallel communication, scheduler pressure, or KV-cache behaviour.

Methods and engine support

vLLM documents support for methods including AWQ, GPTQ, FP8, and INT8. The best method depends on model architecture, accelerator, and available kernels:

▸AWQ protects salient weights and is commonly used for 4-bit serving.
▸GPTQ uses post-training optimization of quantization error.
▸FP8 can be attractive on compatible NVIDIA hardware with optimized kernels.
▸GGUF is a practical ecosystem choice for llama.cpp and edge/CPU use, not a direct replacement for every GPU stack.

The Ramu-DE/vLLM workshop keeps its model in BF16. That is a valid workshop baseline, not evidence that quantized deployments are unsupported or worse.

A safe selection workflow

text

BF16/FP16 baseline
  ↓
measure quality + TTFT + ITL + cache occupancy
  ↓
try one lower-precision candidate
  ↓
re-run the same golden prompts and retrieval/RAG eval suite
  ↓
compare cost per successful answer, not only tokens/sec

Create a quality gate before approving a format change. For example:

▸structured-output validity does not regress;
▸domain-answer accuracy stays within an agreed tolerance;
▸tool-call success stays within tolerance;
▸grounding/faithfulness does not regress for RAG workloads;
▸P95 latency or concurrency improves enough to justify operational complexity.

KV-cache quantization

Weight quantization may let the model fit, while KV cache still limits concurrency. KV-cache quantization can free headroom for more active sequences, but validate long-context quality and attention-sensitive tasks separately. Use the KV-cache worksheet to calculate the potential headroom before assuming it will solve a queueing problem.

Hardware and operational constraints

▸FP8 benefits require compatible hardware and serving kernels.
▸A quantized checkpoint may have a different tokenizer, chat template, or architecture expectation.
▸Fused kernels and tensor parallelism can change numerical and latency behaviour.
▸Keep a rollback path to the prior precision and record the engine/version/config with each benchmark.

Free concepts, Pro decision record

This memo is public so teams avoid choosing precision by headline compression ratio. The Pro serving path creates the decision record: baseline metrics, target constraint, candidate formats, quality gate, observed serving metrics, and rollback plan.