SPEC: Reasoning Models / Extended Thinking

Definition

[Definition] Reasoning models are a class of LLMs that perform extended internal chain-of-thought before producing a final answer — trading increased inference compute and latency for significantly improved accuracy on complex reasoning tasks. Unlike prompting a model to "think step by step," reasoning models generate a hidden internal scratchpad (often thousands of tokens) as an integral part of their inference process.

The Core Paradigm Shift

Standard LLM: token budget fixed, answer generated directly Reasoning Model: token budget is variable; model "thinks" as long as needed before answering

Why Reasoning Models Work

1. ▸More compute at inference = better answers — the "test-time compute scaling" insight
2. ▸Each thinking token is real computation that conditions subsequent predictions
3. ▸The model can explore, backtrack, verify, and self-correct within the thinking block
4. ▸For hard problems, allocating more thinking tokens dramatically improves success rates

Key Reasoning Models (2024–2025)

How They Are Trained

Reasoning models are not just prompted — they are trained differently:

1. ▸Supervised warm-up: fine-tune on examples with explicit reasoning chains (CoT data)
2. ▸Reinforcement Learning: use outcome-based rewards (is the final answer correct?) rather than supervised imitation
   • ▸The model is rewarded for correct answers, not for any specific reasoning format
   • ▸RL allows the model to discover novel reasoning strategies
3. ▸Process Reward Models (PRMs): reward models that evaluate the quality of each reasoning step, not just the final answer

The RL training teaches the model:

• ▸When to think more vs. move on
• ▸How to backtrack from wrong paths
• ▸Verification and self-correction strategies
• ▸Breaking hard problems into sub-problems

Extended Thinking in Claude (Anthropic)

response = client.messages.create(
    model="claude-opus-4-6",
    max_tokens=16000,
    thinking={
        "type": "enabled",
        "budget_tokens": 10000  # max tokens for thinking
    },
    messages=[{"role": "user", "content": "Solve this competition math problem..."}]
)

# Response contains:
# - thinking block (the scratchpad, may be shown or hidden)
# - text block (the final answer)

Test-Time Compute Scaling

[Key Insight] A fundamental insight: you can trade compute for quality at inference time More thinking tokens → better answers (up to a point) This creates a new axis of scaling beyond training: - Training compute scaling: bigger models, more data (expensive, one-time) - Test-time compute scaling: more thinking at inference (cheaper, per-query) For hard problems (competition math, complex code), using 10× more thinking tokens can increase success rate from 30% to 90%.

When Reasoning Models Shine

Latency and Cost Trade-offs

Reasoning models are expensive and slow:

Use reasoning models only when accuracy on hard tasks justifies the cost.

Thinking Tokens Are Hidden Context

• ▸The thinking/scratchpad tokens count against the context window
• ▸They are typically not shown to end users (collapsed or hidden)
• ▸In Claude's API, thinking blocks are clearly separated from the final response
• ▸The model cannot "reopen" thinking after starting its final response

Reasoning vs. Chain-of-Thought Prompting

DeepSeek-R1 (Open Weights)

DeepSeek-R1 is notable as an open-weights reasoning model (MIT license):

• ▸Trained with GRPO (Group Relative Policy Optimization) — a simpler RL variant
• ▸Shows clear <think>...</think> reasoning blocks in output
• ▸Performance competitive with o1 on math/code benchmarks
• ▸Demonstrates RL-based reasoning training is reproducible at lower cost

Related Concepts

• ▸Chain of Thought, Inference, RLHF, Scaling Laws, Test-Time Compute, Benchmarks