SPEC: Evals (Evaluation Frameworks)

Definition

[Definition] Evals (evaluations) are systematic testing frameworks that measure how well an LLM performs on a specific task, use case, or quality dimension. Unlike broad benchmarks that compare models generally, evals are purpose-built for a specific application — they answer "is this model good enough for MY use case?"

Why Evals Are Critical

• ▸Benchmarks measure general capability; evals measure YOUR application's performance
• ▸"Vibes-based" development (it feels right) doesn't scale or catch regressions
• ▸Without evals, you can't know if a model change improved or degraded quality
• ▸Evals enable data-driven model selection and prompt iteration

The Eval Mindset

Types of Evals

1. Automated Evals (Deterministic)

Pass/fail based on exact or programmatic criteria:

• ▸Exact match: model output == expected output
• ▸Regex match: output matches a pattern
• ▸JSON schema validation: output conforms to schema
• ▸Code execution: generated code runs and produces correct output
• ▸Contains check: output contains required phrases/keywords
• ▸API call verification: tool was called with correct parameters

Best for: structured outputs, factual Q&A with ground truth, code generation

2. Model-Based Evals (LLM-as-Judge)

Use another LLM to evaluate output quality:

Best for: subjective quality, format adherence, safety, tone, instruction following

3. Human Evals

Human raters score model outputs:

• ▸Gold standard but expensive and slow
• ▸Best for final validation, not iteration
• ▸A/B preference: "Which response is better, A or B?"
• ▸Annotation platforms: Scale AI, Surge, Prolific

4. Pairwise Comparison Evals

Compare two model outputs head-to-head:

• ▸"Model A response" vs. "Model B response"
• ▸Judge determines winner (or tie)
• ▸Foundation of leaderboards like Chatbot Arena

Key Eval Components

Test Set

• ▸50–500 representative examples from your actual use case
• ▸Diverse coverage of user intents, edge cases, failure modes
• ▸Should include adversarial / tricky examples
• ▸Must NOT be in the model's training data (contamination)

Evaluation Criteria

Define what "good" means explicitly:

Scoring System

• ▸Binary (pass/fail)
• ▸Likert scale (1–5)
• ▸Continuous (0.0–1.0)
• ▸Multi-dimensional (separate scores per criterion)

Eval Frameworks and Tools

The LLM-as-Judge Pattern

[Key Insight] The most scalable eval approach for subjective quality:

JUDGE_PROMPT = """
You are an expert evaluator. Rate the AI response below.

Question: {question}
AI Response: {response}
Reference Answer: {reference}

Score on these dimensions (1-5):
1. Accuracy: Is the information correct?
2. Helpfulness: Does it address the question?
3. Clarity: Is it easy to understand?

Return JSON: {{"accuracy": X, "helpfulness": X, "clarity": X, "reasoning": "..."}}
"""

score = judge_model.complete(JUDGE_PROMPT.format(...))

Best practices:

• ▸Use a stronger model as judge than the model being evaluated
• ▸Include reference answers when available
• ▸Use chain-of-thought for the judge (reasoning before score)
• ▸Use multiple judge models and aggregate to reduce bias
• ▸Validate judge against human labels on a calibration set

Regression Testing

Evals as CI/CD — run before every model/prompt change:

Catches: prompt changes that help one dimension but hurt another, model upgrades that regress on edge cases, fine-tuning that breaks general capabilities.

Eval Anti-Patterns

Practical Eval Workflow

1. ▸Define metrics that map to business outcomes
2. ▸Collect test data from real user interactions
3. ▸Build automated evals for deterministic criteria
4. ▸Add LLM-as-judge for subjective quality
5. ▸Establish baselines before making changes
6. ▸Run evals on every iteration of model/prompt
7. ▸Human spot-check eval failures periodically
8. ▸Monitor production for distribution drift

Related Concepts

• ▸Benchmarks, LLM-as-Judge, Fine-Tuning, Alignment, Hallucination, RAGAS, Observability