benchmarks.site

Language models

Language models: classification, summaries, chat with tools

How good are language models at everyday text jobs: sorting a text into a category, summarizing without losing facts, chatting while using tools?

Key finding

DeepSeek-V4.1-Flash was the most consistent choice across lesson classification, faithful lesson summaries (99.6–99.7% of knowledge items kept, no truncated notes) and streaming tool chat. Abbreviations like EDB and WOS confuse cheap classifiers unless explained in the prompt, and reasoning toggles differ between models.

How to read the numbers

Accuracy and 'facts kept' — higher is better. Time — lower is better.

Hover a column name for its meaning, or see the glossary.

How we tested & caveats · Markdown · CSV

Platform: DeepInfra (unless noted otherwise). Results of single models on specific tasks. Latencies depended on the platform load at the time (queues and timeouts occurred). Prices omitted. Description of metrics and caveats: project overview.

1. Lesson subject classification, 2026-09-22

  • Task: from about 1100–1300 words of a lesson transcript (recognized speech + photo reading), pick the subject from a list of 15 labels (Matematyka, Fizyka, Chemia, Biologia, Geografia, Historia, Język polski, Język angielski, Język niemiecki, Informatyka, WOS, EDB, Religia, Wychowanie fizyczne, Inne – i.e. Mathematics, Physics, Chemistry, Biology, Geography, History, Polish, English, German, Computer science, civics, education for safety, Religion, Physical education, Other), the topic and a confidence. Strict JSON schema enforced, temperature 0.
  • Data: 6 cases – biology, Polish literature, physics (lots of math), mathematics (lots of physics examples), EDB/first aid (ambiguous with biology), an art lesson that should go to „Inne” (Other). About 1.1–2.5k input tokens per case. Small sample: 6 cases, one call per case.
  • "Without explanations" = cases 1–5 without an extra explanation of the abbreviations (EDB, WOS) in the prompt; "with explanations" = 6 cases with a short explanation of the abbreviations.
10 rows · 5 columnsCSV
openai/gpt-oss-20b1.0–1.9 s (one 3.6 s)100% (also without an enforced schema)4/5 (EDB → Biologia)6/6
openai/gpt-oss-120b1.9–3.4 s100%4/5 (EDB → WOS)6/6
deepseek-ai/DeepSeek-V4-Flash1.3–2.8 s (one 8.3 s)100%5/56/6
deepseek-ai/DeepSeek-V4.1-Flash1.8–5 s, one 180 s timeout100%n/a6/6
Qwen/Qwen3-Next-80B-A3B0.9–2.5 s100%4/55/6 (art → „Język polski”, confidence 0.3; in json_object mode it returned a label outside the list: „Sztuka” – "Art")
google/gemma-3-27b-it1.9–4 s100% with schema; without schema wraps the JSON in ```json4/55/6 (art → Język polski)
meta-llama/Llama-4-Scout-17B-16E-Instruct1.2–2.4 s100% with schema; without schema adds a comment4/56/6 (typo „Resusytacja”)
Qwen/Qwen3-32B17–22 s with thinking; 2.1–6.8 s with it disabled100%4/5 (EDB → Wychowanie fizyczne)EDB correct
zai-org/GLM-5.3-Flash6–12 s; later 75 s and a 180 s timeout100%5/5EDB correct
google/gemma-4-26B-A4B-it17–32 s (1.4–4.5 s in a later attempt)100%4/5n/a

Observations: the abbreviations "EDB" and "WOS" confuse cheap models (the first-aid lesson ended up in Biology, WOS or PE); a one-sentence explanation of the abbreviations improved the result for all 8 tested. A strict JSON schema forces a label from the list; without it Qwen3-Next returned „Sztuka”. Gemma-3 and Qwen3-Next put the art lesson in „Język polski” even with the schema. Qwen3-32B requires disabling thinking mode, otherwise it takes 17–22 s to answer.

2. Lesson summary (description), 2026-09-22

  • Task: from the speech transcript and photo reading, write a faithful description of the lesson in Polish. Data: 2 lessons – photosynthesis (2 recordings + a photo reading, about 2.3k input tokens) and „Dziady” part III (a Polish Romantic drama). Qualitative assessment by the author of the measurement, 2 lessons, 1–3 runs.
8 rows · 5 columnsCSV
deepseek-ai/DeepSeek-V4.1-Flash1.35–1.5 s16–37 s570–650best: faithful, complete, no invented facts, clean Polish
deepseek-ai/DeepSeek-V4-Flash1.7 s15–21 sabout 500faithful, more concise
Qwen/Qwen3.5-397B-A17B (thinking disabled)0.8–1.1 s17 sabout 600good; used the term „faza ciemna” ("dark phase") although the teacher advised against it; added one invented line about the next lesson; LaTeX
Qwen/Qwen3-235B-A22B-25071.0–1.2 s30–40 sn/ainvented facts („imię z 44 liter” – "a name of 44 letters", „np. Geniusz stróż” – "e.g. Guardian Genius")
moonshotai/Kimi-K2.6 (default thinking)none79 s3000 reasoning tokens, 0 contentempty result (token limit used up on reasoning)
moonshotai/Kimi-K2.6 (thinking disabled)1.2 s23 sn/agood; one grammatical error
deepseek-ai/DeepSeek-V3.25.3 s103 s (about 7 tokens/s)n/afaithful
google/gemma-4-31B-it3.9 s107 sn/afaithful; LaTeX

3. "Lossless" summaries: how much lesson knowledge survived, 2026-09-29

  • Task: from a 45-minute noisy lesson transcript (fillers, repetitions, dictation with „przecinek” ("comma") spoken aloud, off-topic chatter, broken-off sentences, a whiteboard photo reading with [?]), write a full note that replaces the recording (metaphors, quotes, student questions, homework). The same prompt template for all models (version v2 or v3 depending on the row), one note per lesson. No Google models.
  • Data: 4 synthetic high-school lessons: mathematics (53 knowledge-list items), history (60), biology (57; three overlapping student recordings), Polish (60; a dense lesson, 28 min of speech). Input of 17–28k characters. Each list item is one checkable fact.
  • Scoring: a judge model (DeepSeek-V4-Pro-0813, temperature 0, every verdict with a verbatim quote, checked by code); "present" only when all details are there. The judge is rather too strict (e.g. "missing" for a student question when only the answer is there; "partial" for a surname already garbled in the input transcript).
6 rows · 4 columnsCSV
deepseek-ai/DeepSeek-V4.1-Flash99.6–99.7%407056–99
deepseek-ai/DeepSeek-V4-Pro-081398.7% (v2 and v3)395045–53
moonshotai/Kimi-K2.698.3%450091
MiniMax-M395.7%3850180
deepseek-ai/DeepSeek-V4-Flash93.9% (v2), 90.0% (v3)340086–98
Qwen/Qwen3.5-397B-A17B92.6%4400240
  • DeepSeek-V4.1-Flash, all lessons: 98.7%, 99.6% and 99.7% in three successive versions of the prompt template (2 runs per lesson). 0 truncated notes (finish_reason: length) out of about 100 notes and 0 significant fabrications in 20 checked notes (2 judge flags were false alarms).
  • DeepSeek-V4-Flash loses whole examples (mathematics 77–85%: without the problems with specific sequences and without the derivation of the formula).
  • Items models usually miss: an example shown only on the board, student questions recorded as just the answer.
  • Time with 3 parallel calls 45–65 s, with 8 parallel calls 80–130 s (most calls first got 429 "Model busy").
  • The model fixes mishearings („bazyliszków” → „bazylianów”, "basilisks" → "Basilians"), but a recognition error that changes the meaning (a lost „nie” – "not") is copied faithfully.
  • Limitations: one note per lesson and model, judge instability, one of the Polish-lesson evaluations was interrupted (no funds left on the account, HTTP 402).

DeepSeek-V4-Pro vs V4.1-Flash on lesson description, 2026-11-05

3 lessons (mathematics, biology, history) from the same fixtures, one run each (one or two list items are more than a percentage point). Reasoning disabled (0 reasoning tokens for both).

2 rows · 5 columnsCSV
deepseek-ai/DeepSeek-V4-Pro97.6% (biology 94.7; history 100; mathematics 98.1)97.6%320450–70
deepseek-ai/DeepSeek-V4.1-Flash98.8% (100; 98.3; 100)99.4%369822–26

V4-Pro did not retain more content than V4.1-Flash in this trial. V4-Pro generated about 60 tokens/s.

4. Chat with tools, 2026-09-22

  • Task: two streaming turns with two tools (search the student's notes, and the lesson timetable). Scenario 1: a question that needs both tools at once („jaki mam jutro plan lekcji i co mam w notatkach o rzucie ukośnym z fizyki” – "what is my timetable tomorrow and what do I have in my notes on projectile motion from physics"). Scenario 2: a search that returns no results. All models answered in Polish; none invented note content in the no-results scenario. Small sample: a few calls per model (range in the table).
  • "Arguments" = whether the tool-call arguments are streamed in pieces or arrive whole.
13 rows · 6 columnsCSV
DeepSeek-V4.1-FlashYin pieces1.0–2.3 s / 1.3–2.5 s1.0–1.5 s / 3.1–4.6 snone by default
DeepSeek-V4-FlashYin piecesn/a / 1.1–1.8 s1.7–2.5 s / 5.9–10 snone
Qwen3.5-397B (thinking enabled)Yin pieces0.7 s / 4.2 s3.0 s / 9 sY (separate field)
Qwen3.5-397B (thinking disabled)Yin piecesn/a / 1.0–1.1 s0.8 s / 6.3–7.2 snone
Kimi-K2.6 (default)Yin pieces2.9 s / 11 s (once 115 s / 172 s)28.7 s / 35.9 sY
Kimi-K2.6 (thinking disabled)Yin piecesn/a / 1.1–2.6 s1.0–1.2 s / 5.5–8.4 snone
Kimi-K3Ywhole in one chunk18 s / 27 s1.5 s / 6.8 sY
GLM-5.3Yin pieces1.0–1.1 s / 1.4–1.5 s1.1–1.25 s / 2.7–3.2 sY
DeepSeek-V3.2Ywhole in one chunkn/a / 9.7–19.6 s1.2–1.7 s / 11–28 snone
gpt-oss-120bN (one call per round; 3 rounds)whole in one chunk0.7–0.8 s / 1.5–3.8 s1.1 s / 9.5 sY
DeepSeek-V4-ProYin piecesn/a / 1.2 s3.2 s / 7.7 snone
Qwen3.8-FlashYin pieces1.6 s / 2.7 s4.0 s / 7.4 sY, cannot be disabled
GLM-5.3-FlashYin piecesn/a / 147.6 s182 s / 206 sn/a

Observations:

  • The "search with no results" case: DeepSeek-V4.1-Flash ran two alternative queries, invented nothing and suggested next steps (the best behavior); GLM-5.3 looped through 11 calls over 4 rounds and never answered; Qwen3.8-Flash made 8 calls.
  • Kimi-K3: the first attempt ended with HTTP 429 (engine_overloaded) after 4 retries; a later one worked. GLM-5.3-Flash: 75–206 s to first token and 180 s timeouts. DeepSeek-V4.1-Flash: one 180 s timeout in about 25 calls.
  • GLM-5.3 with thinking disabled "leaks" English reasoning into the answer; it also made a grammatical error („na sprawdzianu”, an incorrect case form for "at the test"). DeepSeek-V4-Pro was verbose (a markdown table, ignored the length limit).
  • Kimi-K2.6 with default thinking used up 3000 tokens on reasoning and returned empty content.
  • Stream format: subsequent tool-call chunks have name, id, type set to null; GLM-5.3 can return two calls in a single chunk; Kimi pads arguments with spaces and assigns IDs like functions.nazwa:0 (nazwa = name); Qwen3.5 and DeepSeek-V3.2 send empty text ("\n\n") before the calls.
  • Reasoning toggles differ: Kimi-K2.6 ignores two alternative disabling parameters; Qwen3-32B requires a different parameter; Qwen3.8-Flash ignores all the ones tried; DeepSeek-V4.1-Flash does not think by default.

Whiteboard photo in chat (1 photo, 1024×1024)

Task: transcribe the problems from the board and solve x²−5x+6=0 and 2x²+3x−2=0. 1 attempt per model.

5 rows · 4 columnsCSV
DeepSeek-V4.1-Flashcorrect1.85 s / 5.0 s720
Qwen3.5-397B (thinking disabled)correct2.5 s / 6.4 s1039
Kimi-K2.6 (default thinking)correct18.3 s / 20.3 s1459
google/gemma-4-31B-itcorrect3.2 s / 14.6 s333
DeepSeek-V4-Flash, Qwen3.8-FlashHTTP 405 – the model does not accept images––

Combining an image with tools worked on DeepSeek-V4.1-Flash (two parallel search calls with a correct query and subject).

5. "Search the web or not" decision in chat, 2026-11-06

35 questions (21 should trigger a search, 14 should not), tools with mock results, one run. Correct decision: DeepSeek-V4.1-Flash 30/35, DeepSeek-V4-Pro-0813 29/35; 0 false searches for both. Typical mistake: the model reached for the materials, got empty results and answered from memory. The result depends on how the instruction is worded, so it is only indicative.

6. External models in chat (Claude, GPT), 2026-11-06: platform behavior

Observations from live calls:

  • Claude (Sonnet and Opus) called through DeepInfra did not report cache hits with a long, repeated request prefix (0 hits).
  • GPT-6 (Responses API): the temperature parameter causes a 400 error; changing the reasoning level between requests invalidated the cache (0 hits despite an identical prefix).
  • The same text (tool definitions) is about 1.9× more tokens for Claude than for GPT-6.