benchmarks.site

Look up a model

How does my model do?

Type a model name to see every test it appears in — our own measurements and public leaderboards — with its place and score. 684 models and services.

Claude Opus 521SimpleBench — #12 · 80.6
CritPt — #25 · 29.1
FrontierMath — #15 · 85.6
OSWorld 2.0 — #1 · 44.33
OSWorld 2.0 — #3 · 36.89
OSWorld 2.0 — #4 · 33.33
OSWorld 2.0 — #5 · 33.01
OSWorld 2.0 — #6 · 31.43
OSWorld 2.0 — #7 · 30.21
OSWorld 2.0 — #8 · 28.98
OSWorld 2.0 — #11 · 25.33
OSWorld 2.0 — #12 · 22.32
OSWorld 2.0 — #14 · 18.81
Vending-Bench 2 — #8 · $11,181.87 ± $2,094
GDPval-AA — #15 · 1729
GDPval-AA — #20 · 1695
Arena (formerly LMArena / Chatbot Arena) — #16 · 1490
Arena (formerly LMArena / Chatbot Arena) — #19 · 1489
F5 Labs CASI & ARS Leaderboard — #5 · 93.05
Epoch Capabilities Index (ECI) — #10 · 164.68
LLM Stats — #11 · 52.5
Claude Fable 5.220ARC-AGI (ARC Prize) — #23 · 81.2
ARC-AGI (ARC Prize) — #24 · 79.3
ARC-AGI (ARC Prize) — #25 · 73.3
SimpleBench — #1 · 94.9
CritPt — #2 · 35.7
CritPt — #3 · 35.4
CritPt — #4 · 32.9
CritPt — #17 · 30.9
CritPt — #24 · 29.4
FrontierMath — #1 · 95.1
GDPval-AA — #6 · 1831
GDPval-AA — #10 · 1751
GDPval-AA — #16 · 1718
AA-Omniscience — #1 · 67
AA-Omniscience — #2 · 65
AA-Omniscience — #3 · 64
Design Arena — #3 · 1367
Epoch Capabilities Index (ECI) — #1 · 173.44
Vals AI — #1 · 71.31
LLM Stats — #1 · 60.8
Gemini 4 Carbon14SimpleBench — #2 · 92.6
CritPt — #1 · 36.0
FrontierMath — #6 · 90.5
MathArena — #1 · 94.28% ± 3.64%
Terminal-Bench — #3 · 64.5% ± 3.2%
SWE Atlas — #4 · 71.45
Vending-Bench 2 — #1 · $16,484.21 ± $1,096
GDPval-AA — #14 · 1733
APEX-Agents — #1 · 84.2% ±4.3%
Arena (formerly LMArena / Chatbot Arena) — #1 · 1526
Design Arena — #8 · 1306
Epoch Capabilities Index (ECI) — #2 · 169.76
Vals AI — #3 · 70.45
LLM Stats — #4 · 57.4
Claude Opus 5.513SimpleBench — #4 · 88.4
CritPt — #7 · 31.7
CritPt — #8 · 31.7
CritPt — #16 · 30.9
FrontierMath — #4 · 91.2
GDPval-AA — #1 · 1866
GDPval-AA — #5 · 1836
GDPval-AA — #19 · 1709
Arena (formerly LMArena / Chatbot Arena) — #4 · 1507
Design Arena — #1 · 1445
Epoch Capabilities Index (ECI) — #4 · 168.54
Vals AI — #4 · 70.14
LLM Stats — #3 · 58.5
GPT-6.1 Sol13SimpleBench — #8 · 82.9
CritPt — #9 · 31.7
CritPt — #10 · 31.7
CritPt — #22 · 30.0
FrontierMath — #2 · 93.7
MathArena — #2 · 92.81% ± 3.10%
Terminal-Bench — #5 · 60.9% ± 3.0%
SWE Atlas — #8 · 65.88
Vending-Bench 2 — #6 · $12,860.61 ± $1,043
Design Arena — #4 · 1366
Epoch Capabilities Index (ECI) — #5 · 168.48
Vals AI — #10 · 63.94
LLM Stats — #12 · 52.2
Claude Sonnet 5.512SimpleBench — #19 · 75.9
CritPt — #12 · 31.4
CritPt — #14 · 31.1
FrontierMath — #10 · 88.8
Vending-Bench 2 — #9 · $11,165.28 ± $1,585
GDPval-AA — #4 · 1838
GDPval-AA — #12 · 1743
Design Arena — #10 · 1290
F5 Labs CASI & ARS Leaderboard — #9 · 91.78
Epoch Capabilities Index (ECI) — #8 · 166.56
Vals AI — #2 · 70.85
LLM Stats — #5 · 56.7
Gemini 4 Argon12SimpleBench — #3 · 91.7
FrontierMath — #13 · 86.0
MathArena — #3 · 84.39% ± 4.65%
Terminal-Bench — #10 · 55.5% ± 2.8%
SWE Atlas — #5 · 70.08
Vending-Bench 2 — #5 · $13,718.16 ± $3,100
APEX-Agents — #2 · 82.4% ±4.3%
MultiChallenge — #2 · 78.70
Arena (formerly LMArena / Chatbot Arena) — #2 · 1522
Epoch Capabilities Index (ECI) — #9 · 166.48
Vals AI — #5 · 69.39
LLM Stats — #9 · 53.3
GPT-6 Astra12SimpleBench — #6 · 86.5
SimpleBench — #7 · 83.6
CritPt — #6 · 31.7
CritPt — #11 · 31.4
FrontierMath — #3 · 93.7
Terminal-Bench — #6 · 60.3% ± 2.8%
SWE Atlas — #6 · 69.42
Vending-Bench 2 — #2 · $15,514.70 ± $1,074
Design Arena — #2 · 1399
Epoch Capabilities Index (ECI) — #3 · 169.0
Vals AI — #8 · 65.02
LLM Stats — #6 · 56.3
Claude Fable 5.111SimpleBench — #5 · 86.6
CritPt — #13 · 31.1
CritPt — #19 · 30.3
CritPt — #23 · 29.7
FrontierMath — #7 · 90.2
GDPval-AA — #9 · 1765
GDPval-AA — #13 · 1741
Arena (formerly LMArena / Chatbot Arena) — #7 · 1502
Design Arena — #6 · 1336
Epoch Capabilities Index (ECI) — #7 · 166.99
LLM Stats — #13 · 52.0
Kimi K3.1 Pro10FrontierMath — #23 · 78.2
MathArena — #5 · 58.27% ± 6.84%
Terminal-Bench — #7 · 59.7% ± 4.2%
SWE Atlas — #14 · 61.11
GDPval-AA — #2 · 1861
Arena (formerly LMArena / Chatbot Arena) — #24 · 1485
Design Arena — #9 · 1297
Epoch Capabilities Index (ECI) — #16 · 161.37
Vals AI — #12 · 62.52
LLM Stats — #7 · 54.4
deepseek-ai/DeepSeek-V4.1-Flash9OCR: handwritten notes, boards and textbook pages — Test from 2026-09-22: 4 photos, 19 models ours
OCR: handwritten notes, boards and textbook pages — "Hard" test from 2026-09-29/30: 14 photos, handwriting a ours
OCR: handwritten notes, boards and textbook pages — Three hard photos of Polish notes, 2026-10-04 ours
OCR: handwritten notes, boards and textbook pages — Textbook pages (print, scans and photos), 2026-10-05 ours
Language models: classification, summaries, chat with tools — Lesson subject classification, 2026-09-22 ours
Language models: classification, summaries, chat with tools — Lesson summary (description), 2026-09-22 ours
Language models: classification, summaries, chat with tools — "Lossless" summaries: how much lesson knowledge survived ours
Language models: classification, summaries, chat with tools — DeepSeek-V4-Pro vs V4.1-Flash on lesson description, 202 ours
Vectara Hallucination Leaderboard — #6 · 4.2 %
gpt-5.58EQ-Bench — #6 · 1295.4
FrontierMath — #16 · 85.3
OSWorld 2.0 — #19 · 13.0
Toolathlon (Tool Decathlon) — #6 · 73.5 ± 1.2
CyberGym — #9 · 81.8%
Polish Linguistic and Cultural Competency (PLCC) — #12 · 95.83
Polish Linguistic and Cultural Competency (PLCC) — #17 · 94.5
Epoch Capabilities Index (ECI) — #20 · 160.1
Claude Haiku 5.57Humanity's Last Exam (HLE) — #13 · 30.1
FrontierMath — #24 · 75.1
MultiChallenge — #10 · 67.67
F5 Labs CASI & ARS Leaderboard — #2 · 94.61
Scale Fortress — #4 · 4
Epoch Capabilities Index (ECI) — #18 · 160.43
Vals AI — #19 · 58.73
Gemini 3.8 Flash7Humanity's Last Exam (HLE) — #5 · 44.5
SimpleBench — #9 · 82.4
SWE Atlas — #19 · 57.33
Remote Labor Index (RLI) — #10 · 5.83
Arena (formerly LMArena / Chatbot Arena) — #9 · 1497
Epoch Capabilities Index (ECI) — #23 · 158.79
Vals AI — #24 · 55.95
google/gemma-4-31B-it7Omni models: video and audio input — Does the model hear the audio from a video file? ours
Omni models: video and audio input — How many frames does the model see? ours
OCR: handwritten notes, boards and textbook pages — Test from 2026-09-22: 4 photos, 19 models ours
Language models: classification, summaries, chat with tools — Lesson summary (description), 2026-09-22 ours
Language models: classification, summaries, chat with tools — Whiteboard photo in chat (1 photo, 1024×1024) ours
Platforms: errors, reasoning billing and prices — Prices from /v1/openai/models (2026-10-16) ours
EuroEval — #19 · 1.67 ± 0.10
GPT-5.6 Sol7SimpleBench — #25 · 71.7
CritPt — #5 · 32.3
FrontierMath — #9 · 89.1
Terminal-Bench — #19 · 38.5% ± 3.8%
OSWorld 2.0 — #9 · 27.34
Arena (formerly LMArena / Chatbot Arena) — #23 · 1485
Epoch Capabilities Index (ECI) — #13 · 162.99
Qwen/Qwen3-VL-235B-A22B-Instruct7OCR: handwritten notes, boards and textbook pages — Test from 2026-09-22: 4 photos, 19 models ours
OCR: handwritten notes, boards and textbook pages — "Hard" test from 2026-09-29/30: 14 photos, handwriting a ours
OCR: handwritten notes, boards and textbook pages — Describing illustrations (charts, diagrams, maps), 2026- ours
OCR: exam-sheet photos with handwritten answers — Blank exam-sheet page ours
OCR: exam-sheet photos with handwritten answers — Cost and time of a single call ours
Reasoning level (reasoning_effort): what models actually do — Reasoning and the output token limit ours
Platforms: errors, reasoning billing and prices — Prices from /v1/openai/models (2026-10-16) ours
Claude Fable 56SimpleBench — #10 · 81.9
FrontierMath — #12 · 87.0
Arena (formerly LMArena / Chatbot Arena) — #5 · 1504
Design Arena — #14 · 1272
Epoch Capabilities Index (ECI) — #11 · 164.13
LLM Stats — #15 · 51.9
deepseek-ai/DeepSeek-V4-Flash6Language models: classification, summaries, chat with tools — Lesson subject classification, 2026-09-22 ours
Language models: classification, summaries, chat with tools — Lesson summary (description), 2026-09-22 ours
Language models: classification, summaries, chat with tools — "Lossless" summaries: how much lesson knowledge survived ours
Language models: short answers, grading text, time and cost — DeepSeek-V4-Flash: undated name vs dated name (2026-10-1 ours
Reasoning level (reasoning_effort): what models actually do — The minimal value on different models (2026-09-03) ours
Reasoning level (reasoning_effort): what models actually do — Cost of reasoning ours
Gemini 4 Hydrogen6Terminal-Bench — #9 · 56.4% ± 3.0%
Vending-Bench 2 — #3 · $14,549.49 ± $495
APEX-Agents — #4 · 78.0% ±4.8%
Arena (formerly LMArena / Chatbot Arena) — #15 · 1490
Design Arena — #15 · 1268
Vals AI — #6 · 68.49
GPT-6 Astra - Provider Adapter6ARC-AGI (ARC Prize) — #1 · 99.9
ARC-AGI (ARC Prize) — #3 · 98.6
ARC-AGI (ARC Prize) — #4 · 98.4
ARC-AGI (ARC Prize) — #5 · 98.4
ARC-AGI (ARC Prize) — #6 · 98.0
ARC-AGI (ARC Prize) — #15 · 96.7
GPT-6 Sol6SimpleBench — #22 · 73.1
CritPt — #15 · 30.9
FrontierMath — #8 · 89.8
Terminal-Bench — #13 · 50.3% ± 3.2%
Vending-Bench 2 — #4 · $14,427.85 ± $1,051
Epoch Capabilities Index (ECI) — #12 · 163.75
GPT-6.1 Terra6SimpleBench — #24 · 72.2
FrontierMath — #5 · 90.9
Terminal-Bench — #12 · 53.3% ± 3.1%
SWE Atlas — #18 · 57.69
Epoch Capabilities Index (ECI) — #6 · 167.67
Vals AI — #14 · 60.54
Grok 4.76Terminal-Bench — #18 · 38.8% ± 3.6%
SWE Atlas — #11 · 62.13
GDPval-AA — #17 · 1718
GDPval-AA — #18 · 1717
Design Arena — #13 · 1273
Vals AI — #20 · 58.62
inclusionAI/Ling-3.0-flash-VL6Omni models: video and audio input — Does the model hear the audio from a video file? ours
Omni models: video and audio input — How many frames does the model see? ours
OCR: handwritten notes, boards and textbook pages — Test from 2026-09-22: 4 photos, 19 models ours
OCR: handwritten notes, boards and textbook pages — Three hard photos of Polish notes, 2026-10-04 ours
OCR: handwritten notes, boards and textbook pages — Textbook pages (print, scans and photos), 2026-10-05 ours
OCR: handwritten notes, boards and textbook pages — Describing illustrations (charts, diagrams, maps), 2026- ours
Inkling 2 Small6MathArena — #7 · 31.88% ± 5.14%
SWE-Bench Pro — #9 · 86.27
SWE Atlas — #13 · 61.89
Design Arena — #7 · 1311
Scale Fortress — #24 · 24
Vals AI — #23 · 56.08
Qwen/Qwen3.5-397B-A17B6OCR: handwritten notes, boards and textbook pages — "Hard" test from 2026-09-29/30: 14 photos, handwriting a ours
OCR: handwritten notes, boards and textbook pages — Three hard photos of Polish notes, 2026-10-04 ours
OCR: handwritten notes, boards and textbook pages — Textbook pages (print, scans and photos), 2026-10-05 ours
OCR: handwritten notes, boards and textbook pages — Describing illustrations (charts, diagrams, maps), 2026- ours
Language models: classification, summaries, chat with tools — Lesson summary (description), 2026-09-22 ours
Language models: classification, summaries, chat with tools — "Lossless" summaries: how much lesson knowledge survived ours
zai-org/GLM-5.3-Flash6Omni models: video and audio input — Does the model hear the audio from a video file? ours
Omni models: video and audio input — How many frames does the model see? ours
OCR: handwritten notes, boards and textbook pages — Test from 2026-09-22: 4 photos, 19 models ours
Language models: classification, summaries, chat with tools — Lesson subject classification, 2026-09-22 ours
EuroEval — #6 · 1.36 ± 0.04
EuroEval — #9 · 1.54 ± 0.08
Claude Fable 5.2 - Provider Adapter5ARC-AGI (ARC Prize) — #7 · 97.7
ARC-AGI (ARC Prize) — #12 · 97.3
ARC-AGI (ARC Prize) — #14 · 96.8
ARC-AGI (ARC Prize) — #19 · 94.9
ARC-AGI (ARC Prize) — #20 · 92.3
Claude Opus 4.75OSWorld 2.0 — #16 · 18.2
OSWorld 2.0 — #18 · 13.9
Vending-Bench 2 — #10 · $10,936.76 ± $1,181
Arena (formerly LMArena / Chatbot Arena) — #8 · 1501
Arena (formerly LMArena / Chatbot Arena) — #12 · 1494
Claude Opus 4.85FrontierMath — #20 · 80.0
OSWorld 2.0 — #13 · 20.6
OSWorld 2.0 — #15 · 18.52
Toolathlon (Tool Decathlon) — #3 · 76.2 ± 3.4
Epoch Capabilities Index (ECI) — #21 · 159.94
gpt-5.45EQ-Bench — #10 · 1257.7
FrontierMath — #22 · 78.6
F5 Labs CASI & ARS Leaderboard — #4 · 93.21
CyberGym — #13 · 79.0%
CyberGym — #18 · 66.3%
GPT-5.5 Pro5SimpleBench — #14 · 76.9
SimpleBench — #15 · 76.9
CritPt — #18 · 30.6
FrontierMath — #11 · 87.7
Epoch Capabilities Index (ECI) — #14 · 162.7
gpt-6-astra5LiveBench — #2 · 84.4
Scale Fortress — #9 · 9
EuroEval — #1 · 1.17 ± 0.05
Polish Linguistic and Cultural Competency (PLCC) — #10 · 96.33
Scale SEAL Leaderboards — #2 · 60.6
GPT-6.1 Sol - Provider Adapter5ARC-AGI (ARC Prize) — #16 · 96.4
ARC-AGI (ARC Prize) — #17 · 96.2
ARC-AGI (ARC Prize) — #18 · 95.0
ARC-AGI (ARC Prize) — #21 · 91.0
ARC-AGI (ARC Prize) — #22 · 82.8
Muse Spark 1.15OSWorld (OSWorld-Verified) — #7 · 80.7
MultiChallenge — #4 · 75.30
Arena (formerly LMArena / Chatbot Arena) — #14 · 1491
Scale Fortress — #10 · 10
CyberGym — #20 · 59.0%
Muse Wonder5FrontierMath — #18 · 82.1
Terminal-Bench — #14 · 47.0% ± 3.8%
GDPval-AA — #8 · 1770
Arena (formerly LMArena / Chatbot Arena) — #3 · 1511
Epoch Capabilities Index (ECI) — #15 · 161.68
Qwen/Qwen3.5-35B-A3B5Omni models: video and audio input — Does the model hear the audio from a video file? ours
Omni models: video and audio input — How many frames does the model see? ours
OCR: handwritten notes, boards and textbook pages — Test from 2026-09-22: 4 photos, 19 models ours
OCR: handwritten notes, boards and textbook pages — "Hard" test from 2026-09-29/30: 14 photos, handwriting a ours
OCR: handwritten notes, boards and textbook pages — Textbook pages (print, scans and photos), 2026-10-05 ours
Qwen/Qwen3.8-27B5OCR: handwritten notes, boards and textbook pages — Test from 2026-09-22: 4 photos, 19 models ours
OCR: handwritten notes, boards and textbook pages — "Hard" test from 2026-09-29/30: 14 photos, handwriting a ours
OCR: handwritten notes, boards and textbook pages — Three hard photos of Polish notes, 2026-10-04 ours
OCR: handwritten notes, boards and textbook pages — Textbook pages (print, scans and photos), 2026-10-05 ours
EuroEval — #17 · 1.67 ± 0.09
claude-opus-4-5-20251101-thinking4Humanity's Last Exam (HLE) — #17 · 25.2
Remote Labor Index (RLI) — #13 · 3.75
MultiChallenge — #18 · 58.97
Scale Fortress — #5 · 5
claude-opus-4-64Humanity's Last Exam (HLE) — #24 · 19.0
EQ-Bench — #17 · 1210.0
Remote Labor Index (RLI) — #12 · 4.17
MultiChallenge — #23 · 56.02
DeepSeek V4.1 Flash 10174Terminal-Bench — #20 · 38.5% ± 3.8%
GDPval-AA — #11 · 1746
Epoch Capabilities Index (ECI) — #22 · 159.41
Vals AI — #13 · 61.26
Fable 5.14Humanity's Last Exam (HLE) — #2 · 46.5
Terminal-Bench — #8 · 57.6% ± 3.3%
SWE Atlas — #3 · 76.45
Remote Labor Index (RLI) — #4 · 17.92
Fable 5.24Terminal-Bench — #2 · 65.8% ± 2.8%
SWE Atlas — #1 · 80.55
APEX-Agents — #3 · 80.1% ±4.7%
Remote Labor Index (RLI) — #1 · 41.25
gemini-3.1-pro-preview4Humanity's Last Exam (HLE) — #3 · 46.4
EQ-Bench — #25 · 1132.3
MultiChallenge — #8 · 71.37
Polish Linguistic and Cultural Competency (PLCC) — #4 · 97
gemini-4-argon4LiveBench — #8 · 77.7
EQ-Bench — #12 · 1236.3
Polish Linguistic and Cultural Competency (PLCC) — #2 · 97.33
Scale SEAL Leaderboards — #4 · 53.2
google/gemma-4-26B-A4B-it4OCR: handwritten notes, boards and textbook pages — Test from 2026-09-22: 4 photos, 19 models ours
Language models: classification, summaries, chat with tools — Lesson subject classification, 2026-09-22 ours
Platforms: errors, reasoning billing and prices — Prices from /v1/openai/models (2026-10-16) ours
Vectara Hallucination Leaderboard — #15 · 5.2 %
GPT-5.4 Pro4SimpleBench — #21 · 74.1
CritPt — #20 · 30.0
FrontierMath — #17 · 82.5
Epoch Capabilities Index (ECI) — #19 · 160.24
gpt-6.1-sol4LiveBench — #6 · 81.4
EQ-Bench — #8 · 1272.4
EuroEval — #2 · 1.18 ± 0.04
Polish Linguistic and Cultural Competency (PLCC) — #6 · 96.83
Kimi K34SWE Atlas — #15 · 60.98
Toolathlon (Tool Decathlon) — #2 · 76.5 ± 1.9
Arena (formerly LMArena / Chatbot Arena) — #20 · 1487
Epoch Capabilities Index (ECI) — #24 · 158.58
moonshotai/Kimi-K2.64OCR: handwritten notes, boards and textbook pages — "Hard" test from 2026-09-29/30: 14 photos, handwriting a ours
OCR: handwritten notes, boards and textbook pages — Textbook pages (print, scans and photos), 2026-10-05 ours
Language models: classification, summaries, chat with tools — Lesson summary (description), 2026-09-22 ours
Language models: classification, summaries, chat with tools — "Lossless" summaries: how much lesson knowledge survived ours
Muse Spark4Humanity's Last Exam (HLE) — #7 · 40.6
MultiChallenge — #3 · 75.52
Arena (formerly LMArena / Chatbot Arena) — #18 · 1489
CyberGym — #23 · 43.5%
Muse Wonder Full4Vending-Bench 2 — #7 · $12,291.56 ± $885
GDPval-AA — #3 · 1857
LLM Stats — #10 · 52.6
LLM Stats — #14 · 52.0
o34Humanity's Last Exam (HLE) — #21 · 20.3
Humanity's Last Exam (HLE) — #23 · 19.2
MMMU / MMMU-Pro — #13 · 76.4
Scale Fortress — #22 · 22
XiaomiMiMo/MiMo-V2.6-Pro4Omni models: video and audio input — Does the model hear the audio from a video file? ours
Omni models: video and audio input — How many frames does the model see? ours
OCR: handwritten notes, boards and textbook pages — "Hard" test from 2026-09-29/30: 14 photos, handwriting a ours
OCR: handwritten notes, boards and textbook pages — Describing illustrations (charts, diagrams, maps), 2026- ours
Claude Opus 4.63Arena (formerly LMArena / Chatbot Arena) — #6 · 1503
Arena (formerly LMArena / Chatbot Arena) — #11 · 1496
CyberGym — #17 · 66.6%
Claude Sonnet 4.63OSWorld 2.0 — #20 · 9.3
OSWorld 2.0 — #21 · 8.3
CyberGym — #19 · 65.2%
claude-opus-53EQ-Bench — #1 · 1364.9
Polish Linguistic and Cultural Competency (PLCC) — #24 · 92.17
Scale SEAL Leaderboards — #6 · 38.6
claude-opus-5-53EQ-Bench — #2 · 1339.5
Scale Fortress — #14 · 14
Scale SEAL Leaderboards — #3 · 55
claude-opus-5.53LiveBench — #5 · 83.4
MathArena — #4 · 82.61% ± 4.72%
Polish Linguistic and Cultural Competency (PLCC) — #21 · 92.5
claude-sonnet-4-63EQ-Bench — #20 · 1196.8
OSWorld (OSWorld-Verified) — #16 · 72.1
EuroEval — #20 · 1.69 ± 0.08
deepseek-ai/DeepSeek-V4-Flash-Vision-Exp3OCR: handwritten notes, boards and textbook pages — Test from 2026-09-22: 4 photos, 19 models ours
OCR: handwritten notes, boards and textbook pages — Describing illustrations (charts, diagrams, maps), 2026- ours
Platforms: errors, reasoning billing and prices — Prices from /v1/openai/models (2026-10-16) ours
deepseek-ai/DeepSeek-V4-Pro3Language models: classification, summaries, chat with tools — DeepSeek-V4-Pro vs V4.1-Flash on lesson description, 202 ours
Reasoning level (reasoning_effort): what models actually do — The minimal value on different models (2026-09-03) ours
Reasoning level (reasoning_effort): what models actually do — Cost of reasoning ours
deepseek-ai/DeepSeek-V4-Pro-08133Language models: classification, summaries, chat with tools — "Lossless" summaries: how much lesson knowledge survived ours
Language models: short answers, grading text, time and cost — DeepSeek-V4-Flash: undated name vs dated name (2026-10-1 ours
Platforms: errors, reasoning billing and prices — Prices from /v1/openai/models (2026-10-16) ours
DeepSeek-V4-Pro3Language models: classification, summaries, chat with tools — Chat with tools, 2026-09-22 ours
EQ-Bench — #22 · 1156.3
CyberGym — #7 · 83.3%
DeepSeek-V4.1-Flash3Language models: classification, summaries, chat with tools — Chat with tools, 2026-09-22 ours
Language models: classification, summaries, chat with tools — Whiteboard photo in chat (1 photo, 1024×1024) ours
Design Arena — #11 · 1288
GBDT 300× d43ML classifiers on TLSH fingerprints of Mach-O files — Main results (out-of-fold, threshold at 0.1% FP) ours
ML classifiers on TLSH fingerprints of Mach-O files — Per-fold thresholds (summed over the 5 test folds) ours
ML classifiers on TLSH fingerprints of Mach-O files — Additional recall figures recorded by the script ours
GBDT 600× d63ML classifiers on TLSH fingerprints of Mach-O files — Main results (out-of-fold, threshold at 0.1% FP) ours
ML classifiers on TLSH fingerprints of Mach-O files — Per-fold thresholds (summed over the 5 test folds) ours
ML classifiers on TLSH fingerprints of Mach-O files — Additional recall figures recorded by the script ours
Gemini 3.7 Flash3Remote Labor Index (RLI) — #11 · 5.00
Arena (formerly LMArena / Chatbot Arena) — #17 · 1489
Epoch Capabilities Index (ECI) — #25 · 158.36
Gemini 4 Argon - Provider Adapter3ARC-AGI (ARC Prize) — #9 · 97.5
ARC-AGI (ARC Prize) — #10 · 97.3
ARC-AGI (ARC Prize) — #11 · 97.3
Gemini 4 Carbon - Provider Adapter3ARC-AGI (ARC Prize) — #2 · 99.1
ARC-AGI (ARC Prize) — #8 · 97.7
ARC-AGI (ARC Prize) — #13 · 96.9
gemini-2.5-pro-preview-06-053Humanity's Last Exam (HLE) — #20 · 21.6
Remote Labor Index (RLI) — #23 · 0.83
Polish Linguistic and Cultural Competency (PLCC) — #22 · 92.17
gemini-3-pro-preview3Humanity's Last Exam (HLE) — #8 · 37.5
Remote Labor Index (RLI) — #22 · 1.25
MultiChallenge — #11 · 65.67
GLM-5.33Language models: classification, summaries, chat with tools — Chat with tools, 2026-09-22 ours
Terminal-Bench — #16 · 41.2% ± 3.3%
CyberGym — #5 · 84.5%
GLM-5.3-Flash3Language models: classification, summaries, chat with tools — Chat with tools, 2026-09-22 ours
LiveBench — #24 · 60.7
Terminal-Bench — #22 · 37.0% ± 3.6%
google/gemini-3.1-pro3OCR: exam-sheet photos with handwritten answers — Blank exam-sheet page ours
OCR: exam-sheet photos with handwritten answers — Cost and time of a single call ours
Platforms: errors, reasoning billing and prices — Prices from /v1/openai/models (2026-10-16) ours
google/gemini-3.5-flash3ASR: Polish speech recognition — Test of 2026-09-22: clean speech synthesis, Polish + Eng ours
OCR: handwritten notes, boards and textbook pages — Test from 2026-09-22: 4 photos, 19 models ours
Platforms: errors, reasoning billing and prices — Prices from /v1/openai/models (2026-10-16) ours
gpt-5-2025-08-073Humanity's Last Exam (HLE) — #15 · 25.3
Humanity's Last Exam (HLE) — #16 · 25.3
Remote Labor Index (RLI) — #20 · 1.67
gpt-5.2-2025-12-113Humanity's Last Exam (HLE) — #14 · 27.8
Remote Labor Index (RLI) — #15 · 2.50
Remote Labor Index (RLI) — #19 · 2.08
gpt-5.4-pro-2026-03-053Humanity's Last Exam (HLE) — #6 · 44.3
MultiChallenge — #9 · 69.23
Scale Fortress — #19 · 19
GPT-5.6 Terra3CritPt — #21 · 30.0
FrontierMath — #14 · 86.0
Epoch Capabilities Index (ECI) — #17 · 160.73
gpt-5.6-sol3EQ-Bench — #13 · 1236.1
EuroEval — #7 · 1.40 ± 0.04
Polish Linguistic and Cultural Competency (PLCC) — #16 · 94.67
gpt-5.6-terra3LiveBench — #18 · 67.2
EQ-Bench — #15 · 1220.1
EuroEval — #13 · 1.58 ± 0.07
GPT-6 Luna3FrontierMath — #21 · 78.9
SWE Atlas — #22 · 50.80
F5 Labs CASI & ARS Leaderboard — #10 · 89.15
gpt-6-sol3Scale Fortress — #15 · 15
EuroEval — #8 · 1.45 ± 0.05
Polish Linguistic and Cultural Competency (PLCC) — #19 · 93.83
Grok 4.63SimpleBench — #18 · 75.9
GDPval-AA — #24 · 1658
CyberGym — #11 · 79.7%
Hy43Terminal-Bench — #21 · 37.9% ± 3.5%
APEX-Agents — #5 · 77.6% ±4.5%
Vals AI — #17 · 59.35
Kimi-K2.63Language models: classification, summaries, chat with tools — Chat with tools, 2026-09-22 ours
Language models: classification, summaries, chat with tools — Whiteboard photo in chat (1 photo, 1024×1024) ours
EQ-Bench — #21 · 1191.4
Kimi-K33Language models: classification, summaries, chat with tools — Chat with tools, 2026-09-22 ours
LiveBench — #20 · 65.0
EQ-Bench — #4 · 1319.2
Logistic regression3ML classifiers on TLSH fingerprints of Mach-O files — Main results (out-of-fold, threshold at 0.1% FP) ours
ML classifiers on TLSH fingerprints of Mach-O files — Per-fold thresholds (summed over the 5 test folds) ours
ML classifiers on TLSH fingerprints of Mach-O files — Additional recall figures recorded by the script ours
MiniMax M33SWE Atlas — #25 · 39.26
OSWorld (OSWorld-Verified) — #11 · 75.2
OSWorld 2.0 — #22 · 4.6
Mistral Large 43Terminal-Bench — #24 · 27.6% ± 3.9%
SWE Atlas — #23 · 45.19
Vals AI — #22 · 56.53
MLP 0.2M3ML classifiers on TLSH fingerprints of Mach-O files — Main results (out-of-fold, threshold at 0.1% FP) ours
ML classifiers on TLSH fingerprints of Mach-O files — Per-fold thresholds (summed over the 5 test folds) ours
ML classifiers on TLSH fingerprints of Mach-O files — Additional recall figures recorded by the script ours
MLP 1M3ML classifiers on TLSH fingerprints of Mach-O files — Main results (out-of-fold, threshold at 0.1% FP) ours
ML classifiers on TLSH fingerprints of Mach-O files — Per-fold thresholds (summed over the 5 test folds) ours
ML classifiers on TLSH fingerprints of Mach-O files — Additional recall figures recorded by the script ours
Muse Ember3SimpleBench — #23 · 72.7
Terminal-Bench — #23 · 30.6% ± 3.8%
Vals AI — #15 · 60.48
Muse Spark 1.23SimpleBench — #20 · 74.5
Toolathlon (Tool Decathlon) — #4 · 75.9 ± 1.3
Arena (formerly LMArena / Chatbot Arena) — #13 · 1493
Muse Spark 1.33SimpleBench — #11 · 81.8
GDPval-AA — #22 · 1681
Arena (formerly LMArena / Chatbot Arena) — #10 · 1497
Muse Spark Open3MathArena — #8 · 27.02% ± 5.61%
SWE Atlas — #24 · 42.28
MultiChallenge — #7 · 74.20
openai/whisper-large-v33ASR: Polish speech recognition — Test of 2026-09-22: clean speech synthesis, Polish + Eng ours
ASR: Polish speech recognition — Degraded recordings, 2026-09-29 (single models, no text ours
ASR: Polish speech recognition — Earlier measurement, 2026-09-23 (one deliberately degrad ours
openai/whisper-large-v3-turbo3ASR: Polish speech recognition — Test of 2026-09-22: clean speech synthesis, Polish + Eng ours
ASR: Polish speech recognition — Degraded recordings, 2026-09-29 (single models, no text ours
ASR: Polish speech recognition — Earlier measurement, 2026-09-23 (one deliberately degrad ours
Opus 5.53Terminal-Bench — #1 · 65.8% ± 3.1%
SWE Atlas — #2 · 77.45
Remote Labor Index (RLI) — #2 · 21.25
Sonnet 5.53Humanity's Last Exam (HLE) — #4 · 46.1
Terminal-Bench — #4 · 63.9% ± 3.0%
SWE Atlas — #7 · 68.28
XiaomiMiMo/MiMo-V2.6-Flash3Omni models: video and audio input — Does the model hear the audio from a video file? ours
Omni models: video and audio input — How many frames does the model see? ours
OCR: handwritten notes, boards and textbook pages — Test from 2026-09-22: 4 photos, 19 models ours
AdaZoom-GUI-4B2ScreenSpot-Pro — #9 · 76.79
ScreenSpot-Pro — #22 · 70.59
anthropic/claude-opus-5-52Reasoning level (reasoning_effort): what models actually do — The minimal value on different models (2026-09-03) ours
Platforms: errors, reasoning billing and prices — Prices from /v1/openai/models (2026-10-16) ours
Claude Mythos Preview2CyberGym — #8 · 83.1%
LLM Stats — #8 · 53.6
Claude Opus 4.52MMMU / MMMU-Pro — #18 · 73.9
CyberGym — #22 · 50.6%
claude-4-opus-thinking2MultiChallenge — #19 · 58.62
Kagi LLM Benchmarking Project — #15 · 74.3
claude-4-sonnet-thinking2MultiChallenge — #21 · 57.11
Kagi LLM Benchmarking Project — #20 · 73.0
claude-fable-52EQ-Bench — #5 · 1312.5
Polish Linguistic and Cultural Competency (PLCC) — #15 · 95
claude-fable-5-22Scale Fortress — #21 · 21
Scale SEAL Leaderboards — #1 · 72.4
claude-fable-5.12LiveBench — #3 · 84.0
Polish Linguistic and Cultural Competency (PLCC) — #14 · 95.17
claude-fable-5.22LiveBench — #1 · 84.9
Polish Linguistic and Cultural Competency (PLCC) — #3 · 97.33
claude-opus-4-1-20250805-thinking2MultiChallenge — #20 · 57.20
Scale Fortress — #18 · 18
claude-opus-4-6-thinking-max2Humanity's Last Exam (HLE) — #11 · 34.4
Scale Fortress — #13 · 13
claude-opus-4-72Humanity's Last Exam (HLE) — #10 · 36.2
EQ-Bench — #7 · 1293.2
claude-sonnet-4-5-20250929-thinking2MultiChallenge — #25 · 55.32
Scale Fortress — #11 · 11
DeepSeek V4.1 Flash2SWE Atlas — #12 · 62.12
Remote Labor Index (RLI) — #7 · 9.58
deepseek-ai/DeepSeek-V3.12Vectara Hallucination Leaderboard — #20 · 5.5 %
CPTU-Bench — #19 · 4.03
deepseek-ai/DeepSeek-V3.22Language models: classification, summaries, chat with tools — Lesson summary (description), 2026-09-22 ours
CPTU-Bench — #15 · 4.14
deepseek-ai/DeepSeek-V4-Flash-07312Language models: short answers, grading text, time and cost — DeepSeek-V4-Flash: undated name vs dated name (2026-10-1 ours
Platforms: errors, reasoning billing and prices — Prices from /v1/openai/models (2026-10-16) ours
DeepSeek-V4-Flash2Language models: classification, summaries, chat with tools — Chat with tools, 2026-09-22 ours
CyberGym — #15 · 76.7%
deepseek-v4.1-flash-10172LiveBench — #13 · 70.3
MathArena — #6 · 42.48% ± 6.32%
Fable 52Terminal-Bench — #15 · 44.8% ± 3.8%
Remote Labor Index (RLI) — #6 · 15.83
Gemini 3 Pro2Arena (formerly LMArena / Chatbot Arena) — #22 · 1486
OmniDocBench — #18 · 92.91
Gemini 3.1 Pro Preview2SimpleBench — #13 · 79.6
Arena (formerly LMArena / Chatbot Arena) — #21 · 1487
Gemini 3.5 Flash2SimpleBench — #16 · 76.7
Toolathlon (Tool Decathlon) — #7 · 67.3 ± 1.2
gemini-4-carbon2LiveBench — #4 · 83.5
Polish Linguistic and Cultural Competency (PLCC) — #1 · 98.33
gemini-4-hydrogen2LiveBench — #9 · 77.0
Polish Linguistic and Cultural Competency (PLCC) — #7 · 96.83
GLM 5.32SWE Atlas — #17 · 58.10
Vals AI — #16 · 59.85
GLM 5.3 Flash2SWE Atlas — #21 · 55.33
Toolathlon (Tool Decathlon) — #1 · 78.4 ± 1.9
glm-5.22EQ-Bench — #18 · 1208.1
CyberGym — #14 · 77.2%
google/gemini-3.1-flash-lite2ASR: Polish speech recognition — Test of 2026-09-22: clean speech synthesis, Polish + Eng ours
OCR: handwritten notes, boards and textbook pages — Test from 2026-09-22: 4 photos, 19 models ours
google/gemma-3-12b-it2OCR: handwritten notes, boards and textbook pages — Test from 2026-09-22: 4 photos, 19 models ours
Vectara Hallucination Leaderboard — #8 · 4.4 %
google/gemma-3-27b-it2OCR: handwritten notes, boards and textbook pages — Test from 2026-09-22: 4 photos, 19 models ours
Language models: classification, summaries, chat with tools — Lesson subject classification, 2026-09-22 ours
GPT 6 Astra2Humanity's Last Exam (HLE) — #1 · 54.8
Remote Labor Index (RLI) — #3 · 20.83
gpt-52Reasoning level (reasoning_effort): what models actually do — The minimal value on different models (2026-09-03) ours
Kagi LLM Benchmarking Project — #21 · 72.7
gpt-5-mini-2025-08-072Humanity's Last Exam (HLE) — #22 · 19.4
EuroEval — #23 · 1.71 ± 0.07
gpt-5-pro-2025-10-062Humanity's Last Exam (HLE) — #12 · 31.6
Scale Fortress — #20 · 20
gpt-5.4-2026-03-052Humanity's Last Exam (HLE) — #9 · 36.2
Polish Linguistic and Cultural Competency (PLCC) — #23 · 92.17
gpt-6-luna2LiveBench — #25 · 60.4
Scale Fortress — #6 · 6
gpt-6.1-terra2LiveBench — #7 · 78.8
Polish Linguistic and Cultural Competency (PLCC) — #18 · 94
gpt-oss-120b2Language models: classification, summaries, chat with tools — Chat with tools, 2026-09-22 ours
Scale Fortress — #3 · 3
GroundCascade2ScreenSpot-Pro — #1 · 85.14
ScreenSpot-Pro — #2 · 85.01
Haiku 5.52Terminal-Bench — #17 · 40.0% ± 3.6%
SWE Atlas — #16 · 59.43
Holo2-235B-A22B2ScreenSpot-Pro — #7 · 78.49
ScreenSpot-Pro — #23 · 70.59
Kimi K2.52OSWorld (OSWorld-Verified) — #24 · 63.3
CyberGym — #25 · 41.3%
kimi-k2.52Humanity's Last Exam (HLE) — #18 · 24.4
MultiChallenge — #15 · 61.39
MAI-UI-32B2ScreenSpot-Pro — #8 · 77.46
ScreenSpot-Pro — #13 · 73.50
meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP82OCR: handwritten notes, boards and textbook pages — Test from 2026-09-22: 4 photos, 19 models ours
CPTU-Bench — #25 · 3.93
meta-llama/Llama-4-Scout-17B-16E-Instruct2OCR: handwritten notes, boards and textbook pages — Test from 2026-09-22: 4 photos, 19 models ours
Language models: classification, summaries, chat with tools — Lesson subject classification, 2026-09-22 ours
mimo-v2.6-pro2LiveBench — #19 · 66.1
GDPval-AA — #21 · 1692
MiniMax-M32Language models: classification, summaries, chat with tools — "Lossless" summaries: how much lesson knowledge survived ours
EQ-Bench — #24 · 1140.1
MiniMaxAI/MiniMax-M32OCR: handwritten notes, boards and textbook pages — "Hard" test from 2026-09-29/30: 14 photos, handwriting a ours
OCR: handwritten notes, boards and textbook pages — Describing illustrations (charts, diagrams, maps), 2026- ours
mistralai/Mistral-Small-3.2-24B-Instruct-25062OCR: handwritten notes, boards and textbook pages — Test from 2026-09-22: 4 photos, 19 models ours
OCR: handwritten notes, boards and textbook pages — Cheaper models, 2026-10-06 (the same 21 readings) ours
moonshotai/Kimi-K32OCR: handwritten notes, boards and textbook pages — Three hard photos of Polish notes, 2026-10-04 ours
Platforms: errors, reasoning billing and prices — Prices from /v1/openai/models (2026-10-16) ours
muse-spark-1.32LiveBench — #16 · 68.6
Polish Linguistic and Cultural Competency (PLCC) — #25 · 91.83
muse-wonder2EQ-Bench — #3 · 1324.9
Polish Linguistic and Cultural Competency (PLCC) — #20 · 93.83
Qwen/Qwen3-32B2Language models: classification, summaries, chat with tools — Lesson subject classification, 2026-09-22 ours
Vectara Hallucination Leaderboard — #25 · 5.9 %
Qwen/Qwen3-ASR-0.6B2ASR: Polish speech recognition — Test of 2026-09-22: clean speech synthesis, Polish + Eng ours
ASR: Polish speech recognition — Degraded recordings, 2026-09-29 (single models, no text ours
Qwen/Qwen3-ASR-1.7B2ASR: Polish speech recognition — Test of 2026-09-22: clean speech synthesis, Polish + Eng ours
ASR: Polish speech recognition — Degraded recordings, 2026-09-29 (single models, no text ours
Qwen/Qwen3-Embedding-0.6B2Embeddings: retrieval accuracy in Polish — Data ours
PL-MTEB (Polish MTEB) — #19 · 65.50
Qwen/Qwen3-Embedding-4B2Embeddings: retrieval accuracy in Polish — Data ours
PL-MTEB (Polish MTEB) — #4 · 72.34
Qwen/Qwen3-Embedding-8B2Embeddings: retrieval accuracy in Polish — Data ours
PL-MTEB (Polish MTEB) — #3 · 73.54
Qwen/Qwen3.5-9B2OCR: handwritten notes, boards and textbook pages — Test from 2026-09-22: 4 photos, 19 models ours
OCR: handwritten notes, boards and textbook pages — Cheaper models, 2026-10-06 (the same 21 readings) ours
Qwen/Qwen3.6-35B-A3B2OCR: handwritten notes, boards and textbook pages — Test from 2026-09-22: 4 photos, 19 models ours
Platforms: errors, reasoning billing and prices — Prices from /v1/openai/models (2026-10-16) ours
Qwen3.5-397B2Language models: classification, summaries, chat with tools — Chat with tools, 2026-09-22 ours
Language models: classification, summaries, chat with tools — Whiteboard photo in chat (1 photo, 1024×1024) ours
Qwen3.8 Max2FrontierMath — #25 · 74.7
GDPval-AA — #23 · 1674
qwen3.8-max2LiveBench — #23 · 60.9
Arena (formerly LMArena / Chatbot Arena) — #25 · 1484
qwen4-max2LiveBench — #11 · 73.3
Design Arena — #5 · 1360
Soniox stt-async-v62ASR: Polish speech recognition — External providers on the same recordings, 2026-11-06 ours
Diarization: who is speaking — Comparison with external providers, 2026-11-06 ours
UiPath Screen Agent w/ Opus 4.52OSWorld (OSWorld-Verified) — #18 · 67.1
OSWorld (OSWorld-Verified) — #21 · 64.4
ABot-OCR1OmniDocBench — #15 · 93.30
agent s3 w/ GPT-5 bBoN1OSWorld (OSWorld-Verified) — #17 · 69.9
agent s3 w/ Opus 4.5 + GPT-5 bBoN1OSWorld (OSWorld-Verified) — #15 · 72.6
agent s3 w/ Opus 4.5 bBoN1OSWorld (OSWorld-Verified) — #19 · 66.0±1.1
ahocorasick_rs1Scanner performance: database loading, re-scans, TLSH in SQL, Aho-Corasick — External libraries ours
ai21labs/jamba-mini-21Vectara Hallucination Leaderboard — #16 · 5.3 %
Alibaba-NLP/gte-Qwen2-1.5B-instruct1PL-MTEB (Polish MTEB) — #12 · 70.00
allenai/olmOCR-2-7B-10251OCR: handwritten notes, boards and textbook pages — Cheaper models, 2026-10-06 (the same 21 readings) ours
amazon/nova-2-lite-v1:01Vectara Hallucination Leaderboard — #12 · 5.1 %
amazon/nova-micro-v1:01Vectara Hallucination Leaderboard — #19 · 5.5 %
amazon/nova-pro-v1:01Vectara Hallucination Leaderboard — #11 · 5.1 %
antgroup/finix_s1_32b1Vectara Hallucination Leaderboard — #1 · 1.8 %
Anthropic Claude Code1SWE-rebench — #7 · 60.4
Anthropic Fable 5 [high]1SWE-rebench — #1 · 64.5
Anthropic Opus 5 [high]1SWE-rebench — #3 · 63.4
Anthropic Sonnet 5 [high]1SWE-rebench — #9 · 56.8
anthropic/claude-fable-51Platforms: errors, reasoning billing and prices — Prices from /v1/openai/models (2026-10-16) ours
anthropic/claude-haiku-5.5-202610071OpenRouter Rankings — #18 · 2779.1
anthropic/claude-opus-4.6 [low]1Context Arena (MRCR) — #25 · 68.8
anthropic/claude-opus-4.6 [medium]1Context Arena (MRCR) — #20 · 72.5
anthropic/claude-opus-4.8 [max]1Context Arena (MRCR) — #16 · 75.2
anthropic/claude-opus-5 [max]1Context Arena (MRCR) — #3 · 91.3
anthropic/claude-opus-5.5-202609211OpenRouter Rankings — #6 · 7365.4
anthropic/claude-sonnet-4.6 [high]1Context Arena (MRCR) — #21 · 72.4
anthropic/claude-sonnet-4.6 [low]1Context Arena (MRCR) — #19 · 72.6
anthropic/claude-sonnet-4.6 [medium]1Context Arena (MRCR) — #22 · 71.7
anthropic/claude-sonnet-5-51Platforms: errors, reasoning billing and prices — Prices from /v1/openai/models (2026-10-16) ours
anthropic/claude-sonnet-5.5-202609281OpenRouter Rankings — #19 · 2666.3
assemblyai/universal-3-5-pro1Open ASR Leaderboard — #15 · 6.21
Aurora1TTS Arena V2 — #1 · 1687
BAAI/bge-m31Embeddings: retrieval accuracy in Polish — Data ours
baidu/ernie-5.5-flash-preview-202611101OpenRouter Rankings — #20 · 2470.8
Basin1CyberGym — #10 · 80.0%
beam-1p0-501b-a23b1Scale Fortress — #7 · 7
Building the automaton in threads1Scanner performance: database loading, re-scans, TLSH in SQL, Aho-Corasick — External libraries ours
ByteDance/Seed-2.0-mini1OCR: handwritten notes, boards and textbook pages — Test from 2026-09-22: 4 photos, 19 models ours
ByteDance/Seed-2.0-pro1OCR: handwritten notes, boards and textbook pages — Describing illustrations (charts, diagrams, maps), 2026- ours
CastleFlow v1.01TTS Arena V2 — #4 · 1558
Chance Vision 1.51MMMU / MMMU-Pro — #1 · 86.9
ChatGPT agent1Remote Labor Index (RLI) — #21 · 1.25
chatgpt-4o1Kagi LLM Benchmarking Project — #14 · 75.0
Claude 3.5 Sonnet1Scale Fortress — #12 · 12
Claude Fable 5.1 Max Effort1F5 Labs CASI & ARS Leaderboard — #7 · 92.30
Claude Mythos 5.21LLM Stats — #2 · 59.3
Claude Opus 4.6 w/ tools1MMMU / MMMU-Pro — #11 · 77.3
Claude Opus 4.6 w/o tools1MMMU / MMMU-Pro — #17 · 73.9
Claude Opus 5.5 Reasoning1F5 Labs CASI & ARS Leaderboard — #8 · 92.00
Claude Sonnet 4.51MMMU / MMMU-Pro — #21 · 68.9
Claude Sonnet 4.6 w/ tools1MMMU / MMMU-Pro — #15 · 75.6
Claude Sonnet 4.6 w/o tools1MMMU / MMMU-Pro — #16 · 74.5
Claude Sonnet 51F5 Labs CASI & ARS Leaderboard — #1 · 98.00
claude_3_5_sonnet_20240620_inspect1METR Time Horizons — #18 · 11.4 min (5.5-22.4)
claude_3_5_sonnet_20241022_inspect1METR Time Horizons — #16 · 20.5 min (10.1-40.8)
claude_3_7_sonnet_inspect1METR Time Horizons — #14 · 60.4 min (33.0-104.2)
claude_3_opus_inspect1METR Time Horizons — #22 · 4.0 min (1.7-8.8)
claude_4_1_opus_inspect1METR Time Horizons — #12 · 100.5 min (59.3-159.5)
claude_4_opus_inspect1METR Time Horizons — #13 · 100.4 min (60.0-163.5)
claude_mythos_preview_early_inspect1METR Time Horizons — #1 · 1044.8 min (508.9-3304.3)
claude_opus_4_5_inspect1METR Time Horizons — #7 · 293.0 min (161.7-623.7)
claude_opus_4_6_inspect1METR Time Horizons — #2 · 718.8 min (316.7-3633.8)
claude-4-5-opus1Kagi LLM Benchmarking Project — #25 · 70.7
claude-4-5-opus-thinking1Kagi LLM Benchmarking Project — #8 · 80.2
claude-4-5-Sonnet1Remote Labor Index (RLI) — #18 · 2.08
claude-4-6-opus1Kagi LLM Benchmarking Project — #23 · 72.4
claude-4-6-opus-thinking1Kagi LLM Benchmarking Project — #5 · 83.6
claude-4-7-opus1Kagi LLM Benchmarking Project — #19 · 73.3
claude-4-7-opus-thinking1Kagi LLM Benchmarking Project — #7 · 80.7
claude-4-8-opus-thinking1Kagi LLM Benchmarking Project — #3 · 88.8
claude-5-fable1Kagi LLM Benchmarking Project — #2 · 88.8
claude-5-fable-thinking1Kagi LLM Benchmarking Project — #1 · 91.4
claude-fable-5_max1Epoch AI Benchmarking Hub — #12 · 87.0
claude-fable-5-11Scale SEAL Leaderboards — #5 · 51.3
claude-fable-5-1_max1Epoch AI Benchmarking Hub — #7 · 90.2
claude-fable-5-2_max1Epoch AI Benchmarking Hub — #1 · 95.1
claude-fable-5-max1MCPMark — #4 · 86.61
claude-fable-5[1m]1OSWorld (OSWorld-Verified) — #2 · 86.0
claude-haiku-5-5_max1Epoch AI Benchmarking Hub — #24 · 75.1
claude-haiku-5.51LiveBench — #14 · 68.9
claude-opus-4-1-202508051Scale Fortress — #23 · 23
claude-opus-4-5-202511011Scale Fortress — #16 · 16
claude-opus-4-81EQ-Bench — #9 · 1267.8
claude-opus-4-8_max1Epoch AI Benchmarking Hub — #20 · 80.0
claude-opus-4-8-max1MCPMark — #6 · 76.38
claude-opus-5_max1Epoch AI Benchmarking Hub — #15 · 85.6
claude-opus-5-5_max1Epoch AI Benchmarking Hub — #4 · 91.2
claude-opus-5-5-max1Scale Fortress — #1 · 1
claude-opus-5[1m]1OSWorld (OSWorld-Verified) — #4 · 83.4
claude-sonnet-4-5-202509291OSWorld (OSWorld-Verified) — #25 · 62.9
claude-sonnet-4-5-20250929#thinking1EuroEval — #15 · 1.56 ± 0.08
claude-sonnet-51EQ-Bench — #14 · 1221.2
claude-sonnet-5-5_max1Epoch AI Benchmarking Hub — #10 · 88.8
claude-sonnet-5.51LiveBench — #10 · 77.0
Coasty CUA v11OSWorld (OSWorld-Verified) — #5 · 82.8
codefuse-ai/F2LLM-v2-0.6B1PL-MTEB (Polish MTEB) — #14 · 69.63
codefuse-ai/F2LLM-v2-1.7B1PL-MTEB (Polish MTEB) — #8 · 72.03
codefuse-ai/F2LLM-v2-14B1PL-MTEB (Polish MTEB) — #1 · 75.13
codefuse-ai/F2LLM-v2-330M1PL-MTEB (Polish MTEB) — #18 · 66.53
codefuse-ai/F2LLM-v2-4B1PL-MTEB (Polish MTEB) — #5 · 73.42
codefuse-ai/F2LLM-v2-8B1PL-MTEB (Polish MTEB) — #2 · 74.61
Codex GPT 5.51Remote Labor Index (RLI) — #9 · 6.25
CohereLabs/cohere-transcribe-03-20261Open ASR Leaderboard — #18 · 6.72
Creation Model1CyberGym — #1 · 90.6%
Cursor Cursor1SWE-rebench — #10 · 51.7
davinci_0021METR Time Horizons — #25 · 0.1 min (0.1-0.2)
Deepdub eTTS 3.21TTS Arena V2 — #14 · 1532
DeepSeek DeepSeek-V4 Pro [high]1SWE-rebench — #14 · 40.2
DeepSeek V4 Pro 08131Toolathlon (Tool Decathlon) — #5 · 74.4 ± 1.2
Deepseek V4p1 Flash 10171MultiChallenge — #5 · 74.90
deepseek-ai/DeepSeek-OCR1OCR: handwritten notes, boards and textbook pages — Cheaper models, 2026-10-06 (the same 21 readings) ours
deepseek-ai/DeepSeek-R11CPTU-Bench — #16 · 4.14
deepseek-ai/DeepSeek-V31CPTU-Bench — #20 · 4.02
deepseek-ai/DeepSeek-V3-03241CPTU-Bench — #18 · 4.03
deepseek-ai/DeepSeek-V3.2-Exp1Vectara Hallucination Leaderboard — #17 · 5.3 %
DeepSeek-OCR 21OmniDocBench — #23 · 90.25
DeepSeek-V3.21Language models: classification, summaries, chat with tools — Chat with tools, 2026-09-22 ours
DeepSeek-V4-Flash, Qwen3.8-Flash1Language models: classification, summaries, chat with tools — Whiteboard photo in chat (1 photo, 1024×1024) ours
DeepSeek-V4-Pro Preview1CyberGym — #21 · 57.7%
deepseek-v4-pro-max1MCPMark — #8 · 71.65
deepseek/deepseek-flash#low1EuroEval — #10 · 1.50 ± 0.05
deepseek/deepseek-flash#no-thinking1EuroEval — #16 · 1.57 ± 0.06
deepseek/deepseek-v4-flash-202607311OpenRouter Rankings — #14 · 3196.4
deepseek/deepseek-v4.1-flash-202609101OpenRouter Rankings — #3 · 15455.0
deepseek/deepseek-v4.1-flash-202610171OpenRouter Rankings — #1 · 61170.1
deepvk/USER-bge-m31PL-MTEB (Polish MTEB) — #25 · 62.46
dots.ocr1OmniDocBench — #21 · 90.77
dots.vlm11MMMU / MMMU-Pro — #20 · 70.1
Duvo Eye-11ScreenSpot-Pro — #17 · 72.93
Edge0/ARK-ASR-0.6B1Open ASR Leaderboard — #21 · 6.75
Eleven Flash v2.51TTS Arena V2 — #18 · 1503
Eleven Multilingual v21TTS Arena V2 — #15 · 1507
Eleven Turbo v2.51TTS Arena V2 — #16 · 1505
Eleven v31TTS Arena V2 — #20 · 1521
elevenlabs/scribe_v21Open ASR Leaderboard — #7 · 5.93
embeddinggemma-300m1Embeddings: retrieval accuracy in Polish — Data ours
Fable 5.1 high1SWE-Bench Pro — #2 · 92.20
Fable-5.11Scale Fortress — #17 · 17
Feyospace-v1.11CyberGym — #6 · 84.4%
FireRed-OCR1OmniDocBench — #16 · 93.26
FluidVoice-Labs/fluid-speech-pro1Open ASR Leaderboard — #11 · 6.06
GBOX Agent1OSWorld (OSWorld-Verified) — #22 · 64.2
Gemini 2.5 Pro 05-061MMMU / MMMU-Pro — #23 · 68.0
Gemini 2.5 Pro Experimental1Humanity's Last Exam (HLE) — #25 · 18.2
Gemini 3 Flash1OmniDocBench — #19 · 92.62
Gemini 3 Pro Preview1SimpleBench — #17 · 76.4
Gemini 3.0 Pro1MMMU / MMMU-Pro — #4 · 81.0
Gemini 3.1 Pro Thinking1MMMU / MMMU-Pro — #5 · 80.5
Gemini 3.8 Flash Cyber1CyberGym — #3 · 86.3%
Gemini 3.8 Flash high1SWE-Bench Pro — #15 · 58.80
Gemini 4 Argon high1SWE-Bench Pro — #3 · 92.16
Gemini Argon 41Remote Labor Index (RLI) — #5 · 17.08
gemini_3_1_pro1METR Time Horizons — #3 · 384.1 min (233.5-694.8)
gemini_3_pro1METR Time Horizons — #8 · 224.3 min (139.6-379.2)
gemini-2.0-flash-0011CPTU-Bench — #8 · 4.29
gemini-2.0-flash-lite-0011CPTU-Bench — #17 · 4.09
gemini-3-pro1Kagi LLM Benchmarking Project — #9 · 80.1
Gemini-3.0-Pro-Preview1Polish Linguistic and Cultural Competency (PLCC) — #11 · 95.83
gemini-3.1-flash-lite-preview1MultiChallenge — #16 · 60.61
Gemini-3.5-Flash1Polish Linguistic and Cultural Competency (PLCC) — #9 · 96.33
Gemini-3.6-Flash1Polish Linguistic and Cultural Competency (PLCC) — #13 · 95.5
Gemini-3.7-Flash1Polish Linguistic and Cultural Competency (PLCC) — #8 · 96.67
Gemini-3.8-Flash1Polish Linguistic and Cultural Competency (PLCC) — #5 · 97
gemini-4-argon_high1Epoch AI Benchmarking Hub — #13 · 86.0
gemini-4-carbon_high1Epoch AI Benchmarking Hub — #6 · 90.5
gemini/gemini-3-flash-preview#thinking1EuroEval — #4 · 1.32 ± 0.04
gemini/gemini-3.1-flash-lite-preview1EuroEval — #12 · 1.59 ± 0.08
gemini/gemini-3.6-flash1EuroEval — #5 · 1.31 ± 0.04
gemini/gemini-3.7-flash1EuroEval — #3 · 1.24 ± 0.03
Gemma 4 26B A4B1MMMU / MMMU-Pro — #19 · 73.8
Gemma 4 31B1MMMU / MMMU-Pro — #12 · 76.9
gladia/solaria-31Open ASR Leaderboard — #24 · 6.86
GLM-51CyberGym — #24 · 43.2%
glm-5-thinking1Kagi LLM Benchmarking Project — #12 · 75.0
GLM-5.11CyberGym — #16 · 68.7%
GLM-5.3 max1SWE-Bench Pro — #11 · 84.30
GLM-OCR1OmniDocBench — #7 · 95.22
google/gemini-2.5-flash-lite1Vectara Hallucination Leaderboard — #3 · 3.3 %
google/gemini-3.6-flash [high]1Context Arena (MRCR) — #15 · 77.3
google/gemini-3.6-flash [medium]1Context Arena (MRCR) — #17 · 74.5
google/gemini-3.7-flash1Platforms: errors, reasoning billing and prices — Prices from /v1/openai/models (2026-10-16) ours
google/gemini-3.7-flash [high]1Context Arena (MRCR) — #4 · 88.6
google/gemini-3.7-flash [low]1Context Arena (MRCR) — #14 · 78.3
google/gemini-3.7-flash [medium]1Context Arena (MRCR) — #8 · 82.1
google/gemma-4-26B-A4B-it non-reasoning1CPTU-Bench — #11 · 4.23
google/gemma-4-26B-A4B-it reasoning1CPTU-Bench — #13 · 4.21
google/gemma-4-31B-it non-reasoning1CPTU-Bench — #5 · 4.32
google/gemma-4-31B-it reasoning1CPTU-Bench — #6 · 4.31
google/gemma-4-31B-it-turbo1OCR: handwritten notes, boards and textbook pages — Test from 2026-09-22: 4 photos, 19 models ours
gpt 6.1 Sol1MultiChallenge — #1 · 81.95
GPT 6.1 Terra1MultiChallenge — #6 · 74.20
gpt_3_5_turbo_instruct1METR Time Horizons — #24 · 0.6 min (0.3-1.1)
gpt_41METR Time Horizons — #21 · 4.0 min (1.9-8.0)
gpt_4_1106_inspect1METR Time Horizons — #20 · 4.0 min (1.9-8.4)
gpt_4_turbo_inspect1METR Time Horizons — #23 · 3.7 min (2.0-6.7)
gpt_4o_inspect1METR Time Horizons — #19 · 7.0 min (4.0-12.9)
gpt_5_1_codex_max_inspect1METR Time Horizons — #9 · 223.7 min (134.3-396.2)
gpt_5_21METR Time Horizons — #4 · 352.2 min (198.1-815.2)
gpt_5_2025_08_07_inspect1METR Time Horizons — #10 · 203.0 min (112.6-405.6)
gpt_5_3_codex1METR Time Horizons — #5 · 349.5 min (194.9-816.4)
gpt_5_41METR Time Horizons — #6 · 341.7 min (186.6-768.8)
gpt-4o-mini-transcribe1ASR: Polish speech recognition — External providers on the same recordings, 2026-11-06 ours
gpt-4o-transcribe1ASR: Polish speech recognition — External providers on the same recordings, 2026-11-06 ours
gpt-4o-transcribe-diarize1ASR: Polish speech recognition — External providers on the same recordings, 2026-11-06 ours
GPT-5 w/ thinking1MMMU / MMMU-Pro — #10 · 78.4
gpt-5-21Kagi LLM Benchmarking Project — #18 · 73.3
gpt-5-51Kagi LLM Benchmarking Project — #4 · 88.8
gpt-5-5-xhigh1MCPMark — #2 · 92.91
gpt-5-6-sol-max1MCPMark — #3 · 92.91
gpt-5-mini-thinking1MultiChallenge — #17 · 58.99
gpt-5-pro1Kagi LLM Benchmarking Project — #11 · 76.8
gpt-5-thinking1MultiChallenge — #13 · 63.19
GPT-5.11MMMU / MMMU-Pro — #14 · 76.0
GPT-5.1 Thinking1MMMU / MMMU-Pro — #9 · 79.0
gpt-5.1-2025-11-13-thinking1MultiChallenge — #12 · 63.41
gpt-5.1-thinking1Humanity's Last Exam (HLE) — #19 · 23.7
GPT-5.2 Thinking w/o Python1MMMU / MMMU-Pro — #7 · 80.4
GPT-5.2 Thinking w/o tools1MMMU / MMMU-Pro — #8 · 79.5
GPT-5.4 Thinking w/ tools1MMMU / MMMU-Pro — #2 · 82.1
GPT-5.4 Thinking w/o tools1MMMU / MMMU-Pro — #3 · 81.2
gpt-5.4-2026-03-05_xhigh1Epoch AI Benchmarking Hub — #22 · 78.6
gpt-5.4-pro-2026-03-05_xhigh1Epoch AI Benchmarking Hub — #17 · 82.5
gpt-5.5_xhigh1Epoch AI Benchmarking Hub — #16 · 85.3
GPT-5.5-Cyber1CyberGym — #4 · 85.6%
gpt-5.5-pro_xhigh1Epoch AI Benchmarking Hub — #11 · 87.7
gpt-5.5-xhigh1Scale Fortress — #25 · 25
gpt-5.61Reasoning level (reasoning_effort): what models actually do — The minimal value on different models (2026-09-03) ours
GPT-5.6 Luna1FrontierMath — #19 · 82.1
GPT-5.6 Sol xhigh1SWE-Bench Pro — #12 · 82.40
GPT-5.6 Terra xhigh1SWE-Bench Pro — #8 · 86.30
gpt-5.6-luna1EQ-Bench — #23 · 1145.9
gpt-5.6-luna_max1Epoch AI Benchmarking Hub — #19 · 82.1
gpt-5.6-sol_max1Epoch AI Benchmarking Hub — #9 · 89.1
gpt-5.6-terra / gpt-5.6-sol1Reasoning level (reasoning_effort): what models actually do — Reasoning and the output token limit ours
gpt-5.6-terra_max1Epoch AI Benchmarking Hub — #14 · 86.0
GPT-6 Astra high1SWE-Bench Pro — #4 · 90.20
gpt-6-astra_max1Epoch AI Benchmarking Hub — #3 · 93.7
gpt-6-luna_max1Epoch AI Benchmarking Hub — #21 · 78.9
gpt-6-sol_max1Epoch AI Benchmarking Hub — #8 · 89.8
gpt-6.1-sol_max1Epoch AI Benchmarking Hub — #2 · 93.7
gpt-6.1-terra_max1Epoch AI Benchmarking Hub — #5 · 90.9
gpt-transcribe1ASR: Polish speech recognition — External providers on the same recordings, 2026-11-06 ours
Gradium TTS1TTS Arena V2 — #10 · 1554
Gradium TTS 2026.101TTS Arena V2 — #9 · 1588
Grok 4.51CyberGym — #12 · 79.0%
Grok 4.7 High1F5 Labs CASI & ARS Leaderboard — #3 · 93.31
Grok Grok 4.5 [high]1SWE-rebench — #2 · 63.8
grok-41Kagi LLM Benchmarking Project — #17 · 73.6
grok-4-20-thinking1Kagi LLM Benchmarking Project — #13 · 75.0
grok-4-51Kagi LLM Benchmarking Project — #6 · 83.5
grok-4.61LiveBench — #22 · 63.5
grok-4.71LiveBench — #17 · 67.9
GTA1 w/ GPT-51OSWorld (OSWorld-Verified) — #23 · 63.4
Haiku 4.5 xhigh1SWE-Bench Pro — #18 · 25.50
Haiku 5.5 high1SWE-Bench Pro — #17 · 52.94
HIPPO Agent w/ Opus 4.51OSWorld (OSWorld-Verified) — #12 · 74.5
HIT-TMG/KaLM-embedding-multilingual-mini-instruct-v11PL-MTEB (Polish MTEB) — #22 · 64.57
HojoAI/Hojo-ASR-V11Open ASR Leaderboard — #19 · 6.74
Holo2-30B-A3B1ScreenSpot-Pro — #11 · 75.21
Holo2-8B1ScreenSpot-Pro — #20 · 71.41
Holo3-35B-A3B1OSWorld (OSWorld-Verified) — #8 · 80.4±2.2
hotchpotch/bekko-embedding-v1-a25m1PL-MTEB (Polish MTEB) — #23 · 61.50
Hume Octave1TTS Arena V2 — #6 · 1551
HunyuanOCR1OmniDocBench — #24 · 89.95
HuzzleWorld-2B1ScreenSpot-Pro — #21 · 70.84
HuzzleWorld-3B1ScreenSpot-Pro — #18 · 72.42
ibm-granite/granite-4.0-h-small1Vectara Hallucination Leaderboard — #14 · 5.2 %
ibm-granite/granite-speech-5.0-2b1Open ASR Leaderboard — #17 · 6.68
Indeed-UI-32B1ScreenSpot-Pro — #15 · 73.31
Indeed-UI-32B-zoomin1ScreenSpot-Pro — #3 · 82.73
Indeed-UI-8B1ScreenSpot-Pro — #14 · 73.37
Indeed-UI-8B-zoom-in1ScreenSpot-Pro — #4 · 80.96
Inkling1EQ-Bench — #16 · 1212.1
Inkling xhigh1SWE-Bench Pro — #16 · 56.90
Inkling, Inkling-Small, gemma-4-31B-it1ASR: Polish speech recognition — Degraded recordings, 2026-09-29 (single models, no text ours
Intelligence-Indeed Agent1OSWorld (OSWorld-Verified) — #1 · 90.2
intfloat/e5-mistral-7b-instruct1PL-MTEB (Polish MTEB) — #11 · 69.94
intfloat/multilingual-e5-large-instruct1PL-MTEB (Polish MTEB) — #10 · 69.13
Inworld TTS1TTS Arena V2 — #7 · 1543
Inworld TTS 1.5 MAX1TTS Arena V2 — #5 · 1565
Inworld TTS MAX1TTS Arena V2 — #3 · 1567
Junie Junie1SWE-rebench — #6 · 61.8
Kimi 2.61OSWorld 2.0 — #23 · 4.6
Kimi K2.61OSWorld (OSWorld-Verified) — #14 · 73.1
kimi-k2-5-thinking1Kagi LLM Benchmarking Project — #10 · 78.5
kimi-k2-61MCPMark — #7 · 72.83
kimi-k2-7-code1MCPMark — #5 · 81.89
kimi-k2-thinking1MultiChallenge — #24 · 55.42
Kimi-K3 max1SWE-Bench Pro — #6 · 88.20
kimi-k3-max1MCPMark — #1 · 96.06
kimi-k3.1-pro1LiveBench — #15 · 68.7
kimi-k3.1-pro-together_max1Epoch AI Benchmarking Hub — #23 · 78.2
Kokoro v1.01TTS Arena V2 — #25 · 1470
KV-Ground-8B + Qwen3.5-27B Consistency Router1ScreenSpot-Pro — #5 · 80.90
KV-Ground-GuiOwl1.5-0315-8B1ScreenSpot-Pro — #16 · 73.18
KV-Ground-GuiOwl1.5-0315-8B-ZoomIn1ScreenSpot-Pro — #6 · 80.52
KV-Ground-GuiOwl1.5-4B-0228-ZoomIn1ScreenSpot-Pro — #10 · 76.41
KV-Ground-Qwen3VL-4B-ZoomIn1ScreenSpot-Pro — #24 · 70.27
Lightning v3.1 Pro1TTS Arena V2 — #13 · 1544
Ling 3.1 Flash 10191GDPval-AA — #25 · 1655
Linq-AI-Research/Linq-Embed-Mistral1PL-MTEB (Polish MTEB) — #6 · 71.10
Logics-Parsing-v21OmniDocBench — #14 · 93.33
MAI-UI-8B1ScreenSpot-Pro — #19 · 71.92
manu/bge-m3-custom-fr1PL-MTEB (Polish MTEB) — #15 · 64.74
Manus 1.01Remote Labor Index (RLI) — #16 · 2.50
Manus 1.51Remote Labor Index (RLI) — #17 · 2.50
Manus_1.61Remote Labor Index (RLI) — #14 · 2.92
meta-llama/Llama-3.3-70B-Instruct-Turbo1Vectara Hallucination Leaderboard — #5 · 4.1 %
meta-models/Muse-Glimmer-30B1OCR: handwritten notes, boards and textbook pages — Describing illustrations (charts, diagrams, maps), 2026- ours
meta-models/Muse-Glimmer-30B reasoning1CPTU-Bench — #1 · 4.42
meta/muse-spark-1.3-contributor-202609021OpenRouter Rankings — #16 · 2964.9
microsoft/azure-speech-09-20261Open ASR Leaderboard — #4 · 5.5
microsoft/harrier-oss-v1-0.6b1PL-MTEB (Polish MTEB) — #7 · 70.47
microsoft/harrier-oss-v1-270m1PL-MTEB (Polish MTEB) — #16 · 66.93
microsoft/Phi-41Vectara Hallucination Leaderboard — #4 · 3.7 %
microsoft/Phi-4-multimodal-instruct1OCR: handwritten notes, boards and textbook pages — Cheaper models, 2026-10-06 (the same 21 readings) ours
MiMo V2.6 Flash1Vals AI — #25 · 54.83
MiMo V2.6 Pro1Vals AI — #21 · 57.81
mimo-v2.5-pro1EQ-Bench — #19 · 1197.7
MiMo-V2.6-Pro and MiMo-V2.6-Flash1ASR: Polish speech recognition — Degraded recordings, 2026-09-29 (single models, no text ours
MinerU-2.51OmniDocBench — #17 · 93.04
MinerU2.5-Pro1OmniDocBench — #6 · 95.75
MiniCPM-OCR 1.01OmniDocBench — #4 · 96.44
MiniMax M3.1 Flash1OSWorld 2.0 — #10 · 26.85
Minimax M3p1 Flash xhigh1SWE-Bench Pro — #10 · 84.31
Minimax MiniMax M31SWE-rebench — #11 · 47.2
MiniMax Speech 02 HD1TTS Arena V2 — #19 · 1503
MiniMax Speech 02 Turbo1TTS Arena V2 — #23 · 1483
MiniMax Speech 2.6 HD1TTS Arena V2 — #21 · 1504
MiniMax Speech 2.6 Turbo1TTS Arena V2 — #17 · 1525
MiniMax Speech 2.8 HD1TTS Arena V2 — #11 · 1545
MiniMax Speech 2.8 Turbo1TTS Arena V2 — #22 · 1502
minimax-m3.1-flash1LiveBench — #21 · 63.7
minimax/minimax-m3.1-flash-202610191OpenRouter Rankings — #11 · 3350.9
mistralai/mistral-large-24111Vectara Hallucination Leaderboard — #9 · 4.5 %
mistralai/Mistral-Large-Instruct-24071CPTU-Bench — #24 · 3.93
mistralai/Mistral-Large-Instruct-24111CPTU-Bench — #21 · 4.00
mistralai/mistral-small-25011Vectara Hallucination Leaderboard — #13 · 5.1 %
mistralai/Mistral-Small-3.1-24B-Instruct-25031OCR: handwritten notes, boards and textbook pages — Cheaper models, 2026-10-06 (the same 21 readings) ours
mistralai/Voxtral-Mini-3B-25071ASR: Polish speech recognition — Test of 2026-09-22: clean speech synthesis, Polish + Eng ours
mistralai/Voxtral-Small-24B-25071ASR: Polish speech recognition — Test of 2026-09-22: clean speech synthesis, Polish + Eng ours
modulate/multilingual1Open ASR Leaderboard — #9 · 5.96
MonkeyOCR V21OmniDocBench — #20 · 91.80
moonshotai/Kimi-K2-Instruct-09051CPTU-Bench — #22 · 3.98
moonshotai/kimi-k3.1-pro-202610231OpenRouter Rankings — #12 · 3241.9
multilingual-e5-large1Embeddings: retrieval accuracy in Polish — Data ours
Musci-research/Musci-ASR-qwen-1019-hf1Open ASR Leaderboard — #5 · 5.7
Muse Ember 1.01SWE Atlas — #20 · 55.99
Muse Spark 1.3 Max1Design Arena — #12 · 1279
Muse Spark Open xhigh1SWE-Bench Pro — #13 · 82.35
Muse Spark Thinking1MMMU / MMMU-Pro — #6 · 80.4
Muse Wonder 1.01SWE Atlas — #9 · 64.83
Muse Wonder 1.0 xhigh1SWE-Bench Pro — #5 · 90.20
muse-ember-1.01Scale Fortress — #8 · 8
muse-spark-1.11EQ-Bench — #11 · 1244.4
muse-spark-open1Scale Fortress — #2 · 2
muse-wonder-1.01LiveBench — #12 · 70.7
muse-wonder-1.0_max1Epoch AI Benchmarking Hub — #18 · 82.1
MuseSpark 1.21OSWorld 2.0 — #17 · 15.84
MuseSpark 1.31OSWorld 2.0 — #2 · 39.25
MVP_Qwen3VL-32B1ScreenSpot-Pro — #12 · 74.15
Nemotron 3.5 ASR Multilingual, language specified1ASR: Polish speech recognition — Live speech (streaming), 2026-11-05/06 ours
Nemotron-Diarization1ASR: Polish speech recognition — Earlier measurement, 2026-09-23 (one deliberately degrad ours
nvidia/Nemotron-3-Diarization1Diarization: who is speaking — Comparison with external providers, 2026-11-06 ours
nvidia/nemotron-3-ultra-550b-a55b-202606041OpenRouter Rankings — #13 · 3198.3
nvidia/Nemotron-3.5-ASR-Streaming-Multilingual-0.6b1ASR: Polish speech recognition — Test of 2026-09-22: clean speech synthesis, Polish + Eng ours
nvidia/nemotron-3.5-super-vl-202610151OpenRouter Rankings — #15 · 2969.1
nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B1Open ASR Leaderboard — #13 · 6.13
nvidia/parakeet-tdt-0.6b-v21Open ASR Leaderboard — #25 · 6.91
o1_inspect1METR Time Horizons — #15 · 38.8 min (21.2-65.0)
o1_preview1METR Time Horizons — #17 · 20.3 min (11.7-33.4)
o3_inspect1METR Time Horizons — #11 · 119.7 min (74.6-190.9)
o3-2025-04-16-reasoning-high1MultiChallenge — #22 · 56.62
o3-pro1Kagi LLM Benchmarking Project — #24 · 72.1
o3-pro-2025-06-10-reasoning-high1MultiChallenge — #14 · 62.40
omarelshehy/arabic-english-sts-matryoshka1PL-MTEB (Polish MTEB) — #20 · 65.07
OpenAI Codex1SWE-rebench — #8 · 58.0
OpenAI gpt-4o-transcribe-diarize1Diarization: who is speaking — Comparison with external providers, 2026-11-06 ours
OpenAI GPT-5.6 Luna [medium]1SWE-rebench — #13 · 43.6
OpenAI GPT-5.6 Sol [medium]1SWE-rebench — #5 · 62.3
OpenAI gpt-live-transcribe1ASR: Polish speech recognition — Live speech (streaming), 2026-11-05/06 ours
OpenAI gpt-realtime-whisper1ASR: Polish speech recognition — Live speech (streaming), 2026-11-05/06 ours
openai/gpt-4.1-2025-04-141Vectara Hallucination Leaderboard — #22 · 5.6 %
openai/gpt-5.4-2026-03-05#none1EuroEval — #25 · 1.76 ± 0.09
openai/gpt-5.4-mini-2026-03-171Vectara Hallucination Leaderboard — #21 · 5.5 %
openai/gpt-5.4-mini-2026-03-17#high1EuroEval — #14 · 1.61 ± 0.05
openai/gpt-5.4-mini-2026-03-17#low1EuroEval — #24 · 1.70 ± 0.06
openai/gpt-5.4-mini-2026-03-17#medium1EuroEval — #22 · 1.64 ± 0.06
openai/gpt-5.4-nano-2026-03-171Vectara Hallucination Leaderboard — #2 · 3.1 %
openai/gpt-5.5 [high]1Context Arena (MRCR) — #12 · 79.3
openai/gpt-5.5 [medium]1Context Arena (MRCR) — #7 · 83.0
openai/gpt-5.5 [xhigh]1Context Arena (MRCR) — #6 · 85.8
openai/gpt-5.6-sol [max]1Context Arena (MRCR) — #1 · 92.4
openai/gpt-5.6-terra [max]1Context Arena (MRCR) — #11 · 80.2
openai/gpt-6-luna-202609221OpenRouter Rankings — #5 · 8081.3
openai/gpt-6.1-sol-202609291OpenRouter Rankings — #10 · 3711.0
openai/gpt-oss-120b1Language models: classification, summaries, chat with tools — Lesson subject classification, 2026-09-22 ours
openai/gpt-oss-20b1Language models: classification, summaries, chat with tools — Lesson subject classification, 2026-09-22 ours
openai/gpt-transcribe1Open ASR Leaderboard — #22 · 6.8
OpenAPA w/ gemini-3.1-pro1OSWorld (OSWorld-Verified) — #9 · 78.3
OpenAudio S21TTS Arena V2 — #12 · 1539
OpenDoc-0.1B1OmniDocBench — #22 · 90.67
Opus 4.81Remote Labor Index (RLI) — #8 · 8.33
Opus 51Terminal-Bench — #11 · 53.3% ± 3.1%
Opus 5 xhigh1SWE-Bench Pro — #1 · 98.00
OrdalieTech/Solon-embeddings-large-0.11PL-MTEB (Polish MTEB) — #17 · 63.49
OS-Symphony w/ GPT-51OSWorld (OSWorld-Verified) — #20 · 65.8
OtterAI/speech-v3.11Open ASR Leaderboard — #2 · 5.21
Ovis2.6-30B-A3B1OmniDocBench — #13 · 93.70
OvisOCR21OmniDocBench — #3 · 96.47
OXR-1.01OmniDocBench — #1 · 97.03
p99lab/audrey-0.71Open ASR Leaderboard — #6 · 5.7
PaddleOCR-VL1OmniDocBench — #9 · 94.18
PaddleOCR-VL-1.51OmniDocBench — #8 · 94.93
PaddleOCR-VL-1.61OmniDocBench — #5 · 96.34
Papla P11TTS Arena V2 — #8 · 1552
Parmesan1TTS Arena V2 — #24 · 1481
Pointer Agent w/ Opus 4.71OSWorld (OSWorld-Verified) — #3 · 83.6
Pointer Agent w/ Sonnet 4.61OSWorld (OSWorld-Verified) — #6 · 81.5
pyahocorasick1Scanner performance: database loading, re-scans, TLSH in SQL, Aho-Corasick — External libraries ours
Qianfan-OCR1OmniDocBench — #12 · 93.90
Qwen 3.7 Plus1OSWorld (OSWorld-Verified) — #13 · 73.3
Qwen 3.7-Plus1OSWorld 2.0 — #24 · 2.8
Qwen 3.8 Max1SWE Atlas — #10 · 64.40
Qwen 3.8 Max 09021Vals AI — #18 · 59.05
Qwen 3.8-Max1F5 Labs CASI & ARS Leaderboard — #6 · 92.31
Qwen 4 Max1Vals AI — #11 · 63.20
Qwen Qwen3.5-35B-A3B1SWE-rebench — #17 · 17.1
Qwen Qwen3.6-27B1SWE-rebench — #15 · 31.2
Qwen Qwen3.6-35B-A3B1SWE-rebench — #16 · 24.7
qwen-3-5-397b1Kagi LLM Benchmarking Project — #16 · 73.7
Qwen/Qwen2.5-72B-Instruct1CPTU-Bench — #23 · 3.95
qwen/qwen3-14b1Vectara Hallucination Leaderboard — #18 · 5.4 %
Qwen/Qwen3-235B-A22B-25071Language models: classification, summaries, chat with tools — Lesson summary (description), 2026-09-22 ours
qwen/qwen3-4b1Vectara Hallucination Leaderboard — #23 · 5.7 %
qwen/qwen3-8b1Vectara Hallucination Leaderboard — #10 · 4.8 %
Qwen/Qwen3-ASR-1.7B-hf1Open ASR Leaderboard — #14 · 6.15
Qwen/Qwen3-ASR-1.7B, raw audio1ASR: Polish speech recognition — Earlier measurement, 2026-09-23 (one deliberately degrad ours
Qwen/Qwen3-Next-80B-A3B1Language models: classification, summaries, chat with tools — Lesson subject classification, 2026-09-22 ours
Qwen/Qwen3-VL-30B-A3B-Instruct1OCR: handwritten notes, boards and textbook pages — Test from 2026-09-22: 4 photos, 19 models ours
Qwen/Qwen3.5-0.8B1OCR: handwritten notes, boards and textbook pages — Cheaper models, 2026-10-06 (the same 21 readings) ours
Qwen/Qwen3.5-122B-A10B non-reasoning1CPTU-Bench — #10 · 4.26
Qwen/Qwen3.5-122B-A10B reasoning1CPTU-Bench — #3 · 4.36
Qwen/Qwen3.5-27B non-thinking1CPTU-Bench — #9 · 4.27
Qwen/Qwen3.5-27B thinking1CPTU-Bench — #4 · 4.34
Qwen/Qwen3.5-2B1OCR: handwritten notes, boards and textbook pages — Cheaper models, 2026-10-06 (the same 21 readings) ours
Qwen/Qwen3.5-35B-A3B non-thinking1CPTU-Bench — #14 · 4.18
Qwen/Qwen3.5-35B-A3B thinking1CPTU-Bench — #12 · 4.22
Qwen/Qwen3.5-35B-A3B-FP81EuroEval — #18 · 1.66 ± 0.09
Qwen/Qwen3.5-4B1OCR: handwritten notes, boards and textbook pages — Cheaper models, 2026-10-06 (the same 21 readings) ours
Qwen/Qwen3.6-27B1OCR: handwritten notes, boards and textbook pages — Describing illustrations (charts, diagrams, maps), 2026- ours
Qwen/Qwen3.6-27B non-reasoning1CPTU-Bench — #7 · 4.29
Qwen/Qwen3.6-27B reasoning1CPTU-Bench — #2 · 4.41
Qwen/Qwen3.6-27B-FP81EuroEval — #11 · 1.60 ± 0.07
qwen/qwen3.6-max-preview [enabled]1Context Arena (MRCR) — #23 · 71.6
qwen/qwen3.8-27b [low]1Context Arena (MRCR) — #9 · 81.8
qwen/qwen3.8-27b [medium]1Context Arena (MRCR) — #13 · 78.9
qwen/qwen3.8-27b [xhigh]1Context Arena (MRCR) — #18 · 74.3
qwen/qwen3.8-max [low]1Context Arena (MRCR) — #10 · 81.5
qwen/qwen3.8-max [medium]1Context Arena (MRCR) — #5 · 87.3
qwen/qwen3.8-max [xhigh]1Context Arena (MRCR) — #2 · 92.3
Qwen3-ASR1ASR: Polish speech recognition — Live speech (streaming), 2026-11-05/06 ours
Qwen3-ASR-1.7B1ASR: Polish speech recognition — Language lessons: Polish and a foreign language in one s ours
qwen3-max1Kagi LLM Benchmarking Project — #22 · 72.5
Qwen3-VL 235B-A22B1MMMU / MMMU-Pro — #22 · 68.1
Qwen3-VL-235B1OmniDocBench — #25 · 89.78
Qwen3.8-Flash1Language models: classification, summaries, chat with tools — Chat with tools, 2026-09-22 ours
qwen3.8-max_xhigh1Epoch AI Benchmarking Hub — #25 · 74.7
Qwen3.8-Max-09021Terminal-Bench — #25 · 27.0% ± 3.9%
Qwen4 Max1GDPval-AA — #7 · 1809
reson8/rumble-21Open ASR Leaderboard — #8 · 5.95
salesforce-sovereign-intelligence/SOV-Transcribe-11Open ASR Leaderboard — #10 · 5.96
Salesforce/SFR-Embedding-2_R1PL-MTEB (Polish MTEB) — #13 · 68.88
Salesforce/SFR-Embedding-Mistral1PL-MTEB (Polish MTEB) — #9 · 69.31
sdadas/mmlw-e5-small1PL-MTEB (Polish MTEB) — #21 · 63.41
Seed 1.5-VL Thinking1MMMU / MMMU-Pro — #24 · 67.6
Seed 1.6-Thinking1MMMU / MMMU-Pro — #25 · 66.4
smallestai/pulse1Open ASR Leaderboard — #16 · 6.57
Snowflake/snowflake-arctic-embed-l-v2.01PL-MTEB (Polish MTEB) — #24 · 61.31
snowflake/snowflake-arctic-instruct1Vectara Hallucination Leaderboard — #7 · 4.3 %
Soniox stt-rt-v31ASR: Polish speech recognition — Live speech (streaming), 2026-11-05/06 ours
Soniox stt-rt-v3, PCM1ASR: Polish speech recognition — Live speech (streaming), 2026-11-05/06 ours
Soniox stt-rt-v61ASR: Polish speech recognition — Live speech (streaming), 2026-11-05/06 ours
Soniox stt-rt-v6, PCM1ASR: Polish speech recognition — Live speech (streaming), 2026-11-05/06 ours
Sonnet 5 xhigh1SWE-Bench Pro — #7 · 88.20
Sonnet 5.5 xhigh1SWE-Bench Pro — #14 · 70.59
sophea/asr-k11Open ASR Leaderboard — #12 · 6.11
sprag/symphony1Open ASR Leaderboard — #23 · 6.85
star-june-20261TTS Arena V2 — #2 · 1633
stepfun/step-5-preview-202610081OpenRouter Rankings — #7 · 6360.0
TeleOCR1OmniDocBench — #2 · 96.91
tencent/hy4-202611041OpenRouter Rankings — #2 · 16042.2
TheStageAI/thewhisper-large-v3-turbo1Open ASR Leaderboard — #20 · 6.74
thinkingmachines/Inkling1OCR: handwritten notes, boards and textbook pages — Describing illustrations (charts, diagrams, maps), 2026- ours
typesafe/jev-1.13-202609171OpenRouter Rankings — #17 · 2868.8
UI-Venus-1-5-30B-A3B1ScreenSpot-Pro — #25 · 69.64
Unlimited-OCR1OmniDocBench — #10 · 94.00
VLAA-GUI w/ Opus 4.51OSWorld (OSWorld-Verified) — #10 · 76.3
Voxtral Mini and Small1ASR: Polish speech recognition — Earlier measurement, 2026-09-23 (one deliberately degrad ours
whisper-large-v31ASR: Polish speech recognition — Language lessons: Polish and a foreign language in one s ours
Wonder1Vals AI — #9 · 64.88
Wonder Full1Vals AI — #7 · 66.79
xai-org/grok-31Vectara Hallucination Leaderboard — #24 · 5.8 %
xai/grok-4-1-fast-reasoning1EuroEval — #21 · 1.69 ± 0.07
XekRung-1.5-27B-Preview1CyberGym — #2 · 88.9%
xiaomi/mimo-v2.6-flash-202609211OpenRouter Rankings — #8 · 4164.6
XiaomiMiMo MiMo V2.5 Pro1SWE-rebench — #12 · 46.5
yoob/yoob-asr-pro-v11Open ASR Leaderboard — #3 · 5.33
Youtu-Parsing1OmniDocBench — #11 · 93.74
z-ai/glm-5.3 [high]1Context Arena (MRCR) — #24 · 69.3
z-ai/glm-5.3-202608161OpenRouter Rankings — #9 · 3738.6
z-ai/glm-5.3-flash-202608261OpenRouter Rankings — #4 · 11806.2
Z.ai GLM-5.2 [high]1SWE-rebench — #4 · 62.9
zoom/scribe_v2.1_pro1Open ASR Leaderboard — #1 · 4.87