SPEC: Benchmarks
Definition
[Definition] Benchmarks are standardized test datasets and evaluation protocols used to measure, compare, and track the capabilities of LLMs across specific tasks or dimensions. They provide a consistent, reproducible way to assess model quality — enabling fair comparisons between models from different organizations.
Why Benchmarks Matter
[Key Insight] - Objective comparison: apples-to-apples comparison across models - Progress tracking: measuring improvement over time - Capability profiling: understand what a model is good and bad at - Model selection: practitioners use benchmarks to choose the right model for their task - Research signal: benchmarks drive the field by defining what to improve
Benchmark Categories
Reasoning and Knowledge
| Benchmark | Measures | Key Details |
|---|---|---|
| MMLU | Broad knowledge across 57 subjects | 14K+ multiple-choice questions |
| MMLU-Pro | Harder MMLU with more options | Fewer guessable answers |
| ARC (Easy/Challenge) | Elementary/high-school science | Grade-school reasoning |
| HellaSwag | Commonsense reasoning (story completion) | 70K+ examples |
| WinoGrande | Commonsense (pronoun resolution) | 44K problems |
| BIG-Bench Hard | Challenging reasoning tasks | Models that GPT-3 failed on |
Math and Quantitative
| Benchmark | Measures | Key Details |
|---|---|---|
| GSM8K | Grade school math word problems | 8.5K problems, requires multi-step |
| MATH | Competition math (AMC/AIME level) | 12.5K problems, very hard |
| MATH-500 | Subset of MATH, standardized | Widely used subset |
| MathBench | Broad math across K-12 to competition | Multi-level coverage |
Coding
| Benchmark | Measures | Key Details |
|---|---|---|
| HumanEval | Python function synthesis | 164 hand-crafted problems |
| MBPP | Python programming from docstrings | 374 problems |
| SWE-bench | Real GitHub issues → patches | Harder, real-world relevance |
| LiveCodeBench | Coding with contamination prevention | Rolling new problems |
Instruction Following and Alignment
| Benchmark | Measures | Key Details |
|---|---|---|
| MT-Bench | Multi-turn instruction following | GPT-4 as judge |
| Alpaca Eval 2.0 | Overall instruction following quality | Human/LLM preference |
| IFEval | Verifiable instruction constraints | "Use exactly N words" etc. |
| Arena (Chatbot Arena) | Human preference (ELO ranking) | Live human judgments |
Factuality and Safety
| Benchmark | Measures | Key Details |
|---|---|---|
| TruthfulQA | Avoidance of false beliefs | 817 adversarial questions |
| HarmBench | Resistance to harmful prompts | 400 behaviors, 7 categories |
| BBQ | Bias in social-group QA | Social stereotype detection |
| BOLD | Bias in open-ended generation | Toxicity + sentiment |
Long Context
| Benchmark | Measures | Key Details |
|---|---|---|
| RULER | Long context retrieval and reasoning | Multiple long-context tasks |
| HELMET | Holistic long context eval | 7 task categories |
| Needle-in-a-Haystack | Information retrieval in long context | Find a specific fact |
| LongBench | Diverse long-document tasks | Multi-lingual |
Multimodal
| Benchmark | Measures | Key Details |
|---|---|---|
| MMMU | College-level multimodal understanding | 11.5K expert-annotated questions |
| DocVQA | Document visual QA | Industrial/business documents |
| ChartQA | Chart comprehension | 9.6K question-answer pairs |
| ScienceQA | Multimodal science QA | K-12 science |
Benchmark Interpretation Caveats
Contamination
- ▸Training data may include benchmark test sets
- ▸Model "memorizes" answers rather than reasoning
- ▸Mitigation: hold-out evaluation sets, LM contamination checks, rolling benchmarks
Overfitting to Benchmarks
- ▸"Goodhart's Law": when a measure becomes a target, it ceases to be a good measure
- ▸Models can be fine-tuned specifically to score well on benchmarks
- ▸High benchmark score ≠ good in production
Single-Number Fallacy
- ▸A single benchmark score hides capability trade-offs
- ▸A model can be #1 on math but terrible at instruction following
- ▸Always look at multi-dimensional profiles
Human Performance Reference
| Benchmark | Human Performance |
|---|---|
| MMLU | ~89% |
| GSM8K | ~100% (trivial for humans) |
| MATH | ~90% (expert mathematicians) |
| HumanEval | ~67% (average programmer) |
Leaderboards
| Resource | Focus |
|---|---|
| Chatbot Arena (LMSYS) | Human preference ELO rankings |
| HuggingFace Open LLM Leaderboard | Open-source model rankings |
| Scale HELM | Holistic evaluation framework |
| LiveBench | Contamination-resistant leaderboard |
How to Use Benchmarks as a Practitioner
- ▸Don't rely on a single benchmark — look at profiles
- ▸Match benchmark to your use case — coding benchmark for coding apps
- ▸Run your own eval — synthetic benchmarks may not reflect your data distribution
- ▸Use human evaluation for subjective tasks
- ▸Track regression — monitor benchmark scores as you fine-tune
- ▸Beware of contamination — prefer newer benchmarks, check test set age
Related Concepts
- ▸Evaluation, Fine-Tuning, Alignment, Hallucination, Grounding, LLM, RLHF