benchmarks.site

Language models

Language models: short answers, grading text, time and cost

Short answers, grading a written answer, and what each call costs and how long it takes.

Key finding

On DeepInfra, the undated name deepseek-ai/DeepSeek-V4-Flash hit an overloaded pool (7/12 requests rejected with HTTP 429), while the dated DeepSeek-V4-Flash-0731 answered every time and faster; automatic prompt caching cut the cost of a long step approx. 3.6×. No grading-accuracy test was run.

How to read the numbers

Time in seconds and cost per call — lower is better.

Hover a column name for its meaning, or see the glossary.

How we tested & caveats · Markdown · CSV

Individual models on text tasks: short answers to questions about a web service (with a few documentation fragments in context), tool calls in a conversation, grading students' answers. There are no grading-accuracy measurements here (agreement of the model's grade with an examiner's) – no such test was carried out. Costs from the provider's estimated_cost field. Caveats: overview.

1. DeepSeek-V4-Flash: undated name vs dated name (2026-10-13)

  • Platform: DeepInfra. Task A: a short "Say OK." (8 input tokens, 3 output tokens). Task B: a step of a conversation with tools, approx. 5.7 thousand input tokens, approx. 30 output tokens (a tool call).
5 rows · 6 columnsCSV
deepseek-ai/DeepSeek-V4-Flash (undated)A127/12 rejected (HTTP 429 engine_overloaded, "Model busy, retry later")8–13 s (those that went through)1.26·10⁻⁶ $
deepseek-ai/DeepSeek-V4-Flash-0731A66/61.2–2.3 s1.02·10⁻⁶ $
deepseek-ai/DeepSeek-V4-Pro-0813A44/40.5–1.3 sn/a
deepseek-ai/DeepSeek-V4-Flash (undated)Bapprox. 8 successfulsome requests rejected as above7.3–15.8 s5.24·10⁻⁴ $ without cache, 1.19·10⁻⁴ $ with a cache hit
deepseek-ai/DeepSeek-V4-Flash-0731Bapprox. 17all went through5.2–12.6 s (typically 6–9 s)3.5·10⁻⁴ $ without cache, 9.8·10⁻⁵ $ with a cache hit (5632 of 5763 tokens from cache)

Observations:

  • The same model under the undated name sat on an overloaded pool: roughly every second request came back with 429, and the same request retried a second later usually went through. The dated version responded every time.
  • 0 reasoning tokens for both names (no reasoning parameter).
  • A repeated long prompt hit the cache automatically; the cost of the step dropped approx. 3.6×.

2. Short answers from documentation (gemma-4-E4B-it, 2026-09-04)

  • Model: google/gemma-4-E4B-it via DeepInfra. Task: answering a question about a web service in Polish, with at most 3 short document fragments in context.
  • Cost: approx. 0.01 gr per question (1 gr = 0.01 PLN; measured; the number of questions in the sample is unknown).
  • The model does not accept photos in this use case (not tested).
  • On 2026-10-16 the model was not on DeepInfra's /models list, but the name was recognized (see Platforms).

3. Grading students' answers and hints: costs

See Reasoning, section 2: DeepSeek-V4-Flash with reasoning 7.3× more expensive than without; DeepSeek-V4-Pro with reasoning approx. 40× (hint) compared with Flash without reasoning; grading a short answer worth 2 points with DeepSeek-V4-Pro: 1914 reasoning tokens, 3.38 gr.

Operational run (2026-09-04): a full mathematics exam sheet graded by a model (DeepSeek-V4-Flash-0731) completed without errors; the score of 14/30 points applies to a specific test paper and is not a measure of accuracy.

4. Models without measurements

  • anthropic/claude-opus-5-5 (DeepInfra): neither quality, cost nor time was measured (on 2026-10-16 the account returned 402). Price from the list in Platforms.
  • Kimi, GLM, text Qwen3.x models, gpt-oss: no measurements in these measurements.