benchmarks.site

Language models

Reasoning level (reasoning_effort): what models actually do

Many models have a 'think harder' setting. Does it actually change anything, and what does it cost?

Key finding

On `minimal`, only DeepSeek-V4-Pro actually produced reasoning tokens; gpt-5 accepts `minimal` but does not reason and gpt-5.6 rejects it. Reasoning is expensive (DeepSeek-V4-Flash 7.3× more with reasoning) and can consume the whole output limit, yielding empty responses (gpt-5.6 at a 2000-token limit).

How to read the numbers

Reasoning tokens = hidden 'thinking' the model is billed for. More tokens = slower and more expensive.

Hover a column name for its meaning, or see the glossary.

How we tested & caveats · Markdown · CSV

Measurements of whether a given reasoning_effort value produces real reasoning tokens, and what it costs. "Reasons" = the response reports a non-zero number of reasoning tokens (reasoning_tokens or a separate field with the reasoning content). Samples are small: a few calls per model, on tasks of grading students' answers and giving hints for exam tasks. Caveats: overview.

1. The minimal value on different models (2026-09-03)

5 rows · 5 columnsCSV
deepseek-ai/DeepSeek-V4-Pro (-0813)DeepInfraYYthe only model in the test that actually reasoned on minimal
deepseek-ai/DeepSeek-V4-Flash (-0731)DeepInfraYNresponse longer than without the parameter, but 0 reasoning tokens; reasons only from low
gpt-5OpenAIYN (0 reasoning tokens)lowest level with real reasoning: low
gpt-5.6 (variants terra, sol)OpenAIN (value rejected)–lowest accepted level: low
anthropic/claude-opus-5-5DeepInfraper Anthropic documentation, N (lowest is low)–not measured live (on 2026-10-16 the account returned 402)

2. Cost of reasoning

3 rows · 5 columnsCSV
deepseek-ai/DeepSeek-V4-Flashhint for an exam task, with reasoning (low) vs without2026-09-03a few callswith reasoning 7.3× more expensive than without
deepseek-ai/DeepSeek-V4-Prothe same task, with reasoning, compared with DeepSeek-V4-Flash without reasoning2026-09-03a few callsapprox. 40× more expensive
deepseek-ai/DeepSeek-V4-Pro (model reconstructed from change history)grading a short open answer worth 2 points, with reasoning2026-09-031 task1914 reasoning tokens, 3.38 gr (1 gr = 0.01 PLN), i.e. roughly the same as grading an essay worth 20 points

Observation: the cost of reasoning hardly depends on the length of the task. A short answer worth 2 points used a similar number of reasoning tokens as a long essay.

3. Reasoning and the output token limit

2 rows · 4 columnsCSV
gpt-5.6-terra / gpt-5.6-sollowest accepted reasoning level, output limit of 2000 tokens, task with a photo2026-09-03empty response: reasoning used up the whole limit before the model wrote a single character; a limit of around 6000–8000 was needed
Qwen/Qwen3-VL-235B-A22B-Instructreasoning low, reading a page from a photo2026-09-13reasoning tokens count toward the same limit as the response, so the limit has to be raised (approx. +1500 tokens was adopted); this is an assumption based on the behavior of other models, an empty response from Qwen was not recorded

Qualitative observation (2026-09-13, no numbers): Qwen3-VL-235B with low reasoning confused smudged boxes and crossed-out answers less often than without reasoning. This was not measured on a set with a reference.

4. Billing of reasoning tokens

With google/gemini-3.1-pro (DeepInfra → Google), reasoning tokens are billed but not included in completion_tokens. The cost calculated from the price list and completion_tokens came out approx. 13× too low compared with estimated_cost (2026-09-17, reading a page from a photo, 5 photos).