benchmarks.site

Vision & OCR

OCR: handwritten notes, boards and textbook pages

Which AI models can read handwritten notes, whiteboards and textbook pages from a photo — including Polish letters, formulas and small numbers?

Key finding

On synthetic Polish notes, handwriting and textbook pages, Kimi-K2.6, Qwen3.5-397B-A17B and DeepSeek-V4.1-Flash were among the most accurate single models, and on print all tested models stayed below 1.5% CER. Small numbers are the weak spot - on one test-schedule page every model "corrected" 2026 to 2025 - and on tiny or distant writing models reconstruct content instead of reading it.

How to read the numbers

CER = share of characters read wrong. Lower is better; 0.02 means about 2 wrong characters in 100.

Hover a column name for its meaning, or see the glossary.

How we tested & caveats · Markdown · CSV

All rows are single models reading one photo. There are no results from combining several readings here. These are general-purpose (vision-language) models; the DeepInfra model list from September 2026 had no dedicated OCR model (olmOCR, dots.ocr, PaddleOCR-VL, DeepSeek-OCR) – see section 3 on redirects. Metrics: CER (lower = better), "diacW" = share of words with Polish diacritics written exactly, "math" = share of passed formula checks (LaTeX), "nums" = share of numbers read correctly (dates, 3.14, 87, 5.18…), facts = share of fact expressions found in the reading. Models with a reasoning mode were measured with reasoning disabled unless noted otherwise. All photos are synthetic (handwriting from fonts + simulated degradation); real notebooks will be harder.

1. Test from 2026-09-22: 4 photos, 19 models

  • Platform: DeepInfra (Google models – through DeepInfra to Google; ByteDance Seed – upstream ByteDance/BytePlus).
  • Data: 4 images. (1) notes_clean – Caveat font on squared paper, 1200×1600; (2) notes_photo – the same with rotation, perspective, shadow, blur, noise, contrast ×0.7, JPEG q60, 1320×1720; (3) board_photo – Kalam font on a whiteboard with strong glare, 2 columns, JPEG q70; (4) cursive_photo – Marck Script cursive handwriting, denser, JPEG q60, 1300×1700 (the hardest). Content: Polish diacritics, the pangram „Zażółć gęślą jaźń / źdźbła łódź” (a Polish pangram packed with diacritics), x=(−b±√Δ)/2a, E=mc², H₂SO₄, reactions, limits, derivatives, dates, page numbers.
  • 251 calls. CER computed on the main text of images 1, 2 and 4. Number of checks: 14 (notes), 9 (whiteboard), 11 (cursive). "date" = whether the small date on the degraded photo was read as 22.09.2026. "seq" = total latency / time to first token, streaming, one request at a time; "conc" = median latency with about 10 parallel requests; "inTok" = input tokens for the 1320×1720 photo (with a short instruction).
22 rows · 9 columnsCSV
google/gemini-3.1-flash-lite0.0181.001.000.97N (2025)4.3 / 2.84.21305
ByteDance/Seed-2.0-mini0.0190.990.981.00Y10.9 / 4.97.41515
google/gemma-4-31B-it-turbo0.0190.990.980.89N (2020)48–112 / 3–721.6490
google/gemma-4-31B-it0.0190.990.980.89N (2023)n/a22.8490
Qwen/Qwen3.6-35B-A3B0.0220.991.000.89Y80 / 5558.81169
Qwen/Qwen3.5-35B-A3B0.0250.961.000.89Y6.4 / 2.14.31170
deepseek-ai/DeepSeek-V4.1-Flash0.0260.971.000.89N (2025)9.5 / 2.69.81189
inclusionAI/Ling-3.0-flash-VL0.0280.931.000.89N (2025)8.7 / 2.27.92446
Qwen/Qwen3-VL-235B-A22B-Instruct0.0290.951.000.89N (2025)32 / 2.742.72430
XiaomiMiMo/MiMo-V2.6-Flash0.0330.930.980.87skipped4.7 / 2.14.62430
Qwen/Qwen3.8-27B0.0340.890.980.83Nn/a26.12438
deepseek-ai/DeepSeek-V4-Flash-Vision-Exp0.0380.890.950.88Yn/a15.7549
Qwen/Qwen3-VL-30B-A3B-Instruct0.0450.921.000.89N (2025)43 / 2.322.32430
google/gemma-3-27b-it0.0470.920.870.81Y27 / 2.817.8472
meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP80.0560.840.960.87skipped5.9 / 2.16.92677
meta-llama/Llama-4-Scout-17B-16E-Instruct0.0630.820.930.82skippedn/a7.92677
mistralai/Mistral-Small-3.2-24B-Instruct-25060.0650.810.940.85skippedn/a8.52631
google/gemma-3-12b-it0.0780.910.910.82N (22.07)n/a15.1472
Qwen/Qwen3.5-9B (reasoning not disabled)0.3690.700.790.77Yn/a471167
google/gemini-3.5-flash (reference, cursive only)0.0231.001.001.00–n/a5.51305
google/gemma-4-26B-A4B-it0.0231.000.910.83–n/a98–180 (2 of 4 timed out)490
zai-org/GLM-5.3-Flash–––––all 4 calls with a photo timed out (180–400 s)––

Observations:

  • The word „źdźbła” ("blades [of grass]") was missed most often (most models "normalized" it to żdźbła/zdźbła); read correctly by, among others, gemini-3.1-flash-lite.
  • The small date and small numbers on the degraded photo are unreliable: the year 2026 was read as 2025 (gemini-3.1-flash-lite, Qwen3-VL-235B/30B, DeepSeek-V4.1-Flash, Ling, Qwen3.8), as 2020/2023 (gemma-4), and Llama-4, Mistral and MiMo skipped the date. In the cursive text „5.18” often became „518”. Only these read the date correctly: Seed-2.0-mini, Qwen3.5-35B, Qwen3.6-35B, gemma-3-27b, DeepSeek-V4-Flash-Vision-Exp.
  • Models "fill in from memory": Llama and Mistral invented Polish words („zdębła”, „jazın”, „gęsta”); on the whiteboard with glare all the top models reproduced the formula „sin²α+cos²α=1”, probably from knowledge rather than from reading.
  • Qwen3-VL-235B skipped the small line with the date on the cursive page. Seed-2.0-mini added a heading „### Prawa matematyczne” ("Mathematical laws").
  • Generation speed: Qwen3-VL-235B/30B about 12–15 tokens/s (30–45 s per page); gemma-4-31B-it-turbo 48–112 s; Qwen3.6-35B-A3B 48–62 s to first token; gemma-4-26B-A4B 98–180 s; Qwen3.5-9B with reasoning enabled hit the token limit (150 s, CER 0.37).
  • The number of image tokens depends on the model: constant for Gemini (about 1305) and Seed (about 1515), saturating for Qwen3.5-35B (about 1170) and DeepSeek-V4.1-Flash (about 1189), constant and low for gemma-3/4 (about 256 – they lose small digits and subscripts), growing with resolution for Qwen3-VL, MiMo, Ling (Qwen3-VL-235B: 13884 tokens for 3276×4284 and 16568 for 7800×10150).
  • Formats and limits (platform behavior): MIME image/jpg and image/heic – HTTP 422; HEIC bytes labeled as jpeg – HTTP 400 on models hosted by DeepInfra (Gemini and Seed accept them); WebP – 200; a 21.8 MB PNG (29 MB request) accepted by Gemini, Qwen, gemma, rejected by Seed (10 MiB limit); a 79 Mpx image accepted by Gemini, Qwen3.5, gemma and Qwen3-VL-235B, rejected by Seed (36 Mpx limit). A rotated photo (EXIF 6) was read correctly by gemini, DeepSeek-V4.1, Seed, Qwen3.5, MiMo; Qwen3-VL-235B had diacW 0.81 on it.
  • JSON output can be dangerous for formulas: gemini-3.1-flash-lite wrote \neq without doubling the backslash, so after parsing it came out as a newline character + "eq" (3 of 3 attempts on notes_photo, also with a strict JSON schema); gemma-3-27b broke \frac, \times, \rightarrow. Markdown with a YAML header parsed 6/6 without damage. The instruction "return JSON" alone, without enforcing the format, gave valid JSON in 12 of 22 attempts; with enforcement (json_object) 21 of 22 (Ling once looped until the token limit); Seed-2.0-mini returned HTTP 500 with a strict JSON schema.
  • Image scaling (cursive page): capping the longer side at 2048 px did not lower the score for any model, and it prevented the growth in token count for Qwen3-VL, MiMo, Ling (3–10×).

2. "Hard" test from 2026-09-29/30: 14 photos, handwriting and small numbers

  • Platform: DeepInfra.
  • Data: 13 pages, each in a normalne (normal) and a trudne (hard) version (letters drawn separately with random slant, wavy paper, low resolution, strong blur and noise, JPEG 40–50). Pages: squared notebook (EXIF 6), lined notebook in cursive in poor light, chemistry with a table, whiteboard shot from the side with glare and a force diagram, chalkboard, printed worksheet with answers in pen, slide with integrals, PL/EN vocabulary (EXIF 8, motion blur), dense long page, cell diagram (EXIF 3), physics problems, test schedule, whiteboard with stoichiometry. The trudne set: 10 pages (105 facts; 210 over 2 runs) and separately 4 pages with small numbers (dense page, physics problems, schedule, stoichiometry). Rows from 2026-09-29: one run; the "separate measurement" row: 2 runs.
  • Facts: 135 expressions in the photos (total for the set).
9 rows · 6 columnsCSV
deepseek-ai/DeepSeek-V4.1-Flash4.1%95.2%5.7% / 84%5–112026-09-29
Qwen/Qwen3.5-35B-A3B3.1%95.2%2.5% / 94%42026-09-29
moonshotai/Kimi-K2.62.5%97.1%2.3% / 91%11–192026-09-29
moonshotai/Kimi-K2.6 (separate measurement, 2 runs)2.8%96.7% (203/210)1.6% / 92.0%11.42026-09-30
Qwen/Qwen3.5-397B-A17B2.8%94.3%1.1% / 86%7–92026-09-29
XiaomiMiMo/MiMo-V2.6-Pro2.7%95.2%3.5% / 86%7–82026-09-29
Qwen/Qwen3-VL-235B-A22B-Instruct3.4%95.2%2.4% / 91%16–262026-09-29
Qwen/Qwen3.8-27B3.4%92.4%n/a72026-09-29
MiniMaxAI/MiniMax-M36.4%92.4%n/a102026-09-29

On the normalne pages DeepSeek-V4.1-Flash read 8 of 10 pages without error (CER 2.2%, 104/105 facts). Differences between models show up mainly on the pages with small numbers in the trudne version.

The most dangerous errors: on small handwriting DeepSeek-V4.1-Flash added lines that do not exist („Kartkówka: wtorek 14:05” – "Quiz: Tuesday 14:05" – instead of „Konsultacje: wtorek 14:30” – "Office hours: Tuesday 14:30") and recalculated an example by itself („r = 9”, „a₇ = 762”). All single models "corrected" the year 2026 to 2025 on the test-schedule page („05.11.2025”, „2025/2027”).

Photo of a whole classroom with the board in the distance (2026-10-19, 1 photo)

A board in the distance (about 8 px per character) read independently by three models: Kimi-K2.6 read formulas with an absolute value |x|; Qwen3.5-397B and Qwen3.5-35B "read" formulas for complex numbers that were not on the board; Qwen3.5-35B twice looped until the 6144-token limit. Agreement between the models' readings (trigram overlap): 0.95–0.99 for three pages of the trudne set, 0.33–0.41 for this distant photo. Conclusion: for small writing seen from a distance, models reconstruct the content instead of reading it. This is a single photo.

Three hard photos of Polish notes, 2026-10-04

Synthetic fixtures, 3 photos, one run.

5 rows · 4 columnsCSV
deepseek-ai/DeepSeek-V4.1-Flash4.5%90%about 5 s
inclusionAI/Ling-3.0-flash-VL7.8%82.5%25–265 s
Qwen/Qwen3.5-397B-A17B (low reasoning)1.5–1.9%88–92%1–4 min
Qwen/Qwen3.8-27B5.3–5.8%85–90%about 20 s
moonshotai/Kimi-K3 (variant described in the source as "off")0.4%100%n/a

3. Textbook pages (print, scans and photos), 2026-10-05

  • Platform: DeepInfra.
  • Data: 7 "textbook-like" pages (history, quadratic function with formulas, solubility table with a reaction, physics in two columns with a figure, a poem, two dense pages of 2000 characters – biology and sequences), each in 3 versions: 150 dpi scan, phone photo (skewed, shadow, noise, JPEG 72), "hard" photo (1150 px, shadow across half the page, JPEG 55). 21 readings per model, 56 facts (numbers, formulas, table cells, reaction). The source gives results without confidence intervals.
6 rows · 8 columnsCSV
deepseek-ai/DeepSeek-V4.1-Flash0.10%0.13%0.38%99%100%3/36.8
inclusionAI/Ling-3.0-flash-VL0.08%0.11%0.26%100%100%3/38.0
Qwen/Qwen3.5-35B-A3B1.39%1.27%0.73%98%100%3/35.4
Qwen/Qwen3.8-27B0.22%0.20%0.36%96%100%3/312.1
Qwen/Qwen3.5-397B-A17B0.03%0.06%0.00%99%100%3/312.6
moonshotai/Kimi-K2.60.04%0.04%0.06%100%100%3/330.2

On print all models are close to each other (CER below 1.5%, page number 100%). On hard handwriting (the notebook photos from 04.10, section 2) DeepSeek-V4.1-Flash was clearly better than Ling (CER 4.5% vs 7.8%).

Cheaper models, 2026-10-06 (the same 21 readings)

Note: DeepInfra silently redirects retired models to a replacement; the "responded" column is the model field from the response.

9 rows · 7 columnsCSV
Qwen/Qwen3.5-0.8BQwen/Qwen3.5-9B0.31%0.64%0.51%98%90%
Qwen/Qwen3.5-2BQwen/Qwen3.5-9B0.31%0.61%0.53%98%90%
Qwen/Qwen3.5-4BQwen/Qwen3.5-9B0.31%0.64%0.53%98%90%
Qwen/Qwen3.5-9BQwen/Qwen3.5-9B0.31%0.64%0.53%98%90%
deepseek-ai/DeepSeek-OCRgoogle/gemma-4-31B-it0.55%0.48%1.12%96%100%
allenai/olmOCR-2-7B-1025google/gemma-4-31B-it0.55%0.73%1.01%96%100%
microsoft/Phi-4-multimodal-instructgoogle/gemma-3-12b-it100%100%100%0%0%
mistralai/Mistral-Small-3.1-24B-Instruct-2503Mistral-Small-3.2-24B0.95%0.98%0.91%99%100%
mistralai/Mistral-Small-3.2-24B-Instruct-2506(the same)0.30%0.98%0.94%98%100%

Conclusions: "DeepSeek-OCR" and "olmOCR" on DeepInfra (as of 2026-10-06) are in fact gemma-4-31B-it (retired, with replaced_by); Qwen3.5-0.8B/2B/4B were answered by the 9B model (identical results), which got the page number wrong in 10% of readings; Mistral-Small had about 1% CER on photos.

305-page scan and loops on leader dots, 2026-10-08

Test pages like those of a WOS textbook (civics; two columns, boxes, a chart, photos, a dense page of ~4000 characters, a table of contents with leader dots, exercises with dotted lines), A4 scan at 300 dpi, real calls, small sample.

3 rows · 3 columnsCSV
regular (701–1467 output tokens)8–62 s, sometimes 100–185 s (platform queue)7–27 s
exercises with dotted lines8–100 s, passesresponse judged as looped (15 identical lines of dots)
table of contents with dotsloop „. . . .” until the 6144-token limit (88–149 s), 3/3 attemptsloop until 6144 tokens in one of the attempts (older prompt), 490 tokens in another

Ling: in this measurement it had no flex mode (the platform responded with service_tier: default).

4. Describing illustrations (charts, diagrams, maps), 2026-10-06

  • Platform: DeepInfra.
  • Data: 9 drawings with deliberately made-up data (so that the model has to actually read them): a bar chart with values, a function graph with points without labeled coordinates, a circuit diagram, a photosynthesis diagram, a triangle with an altitude, a map of an island with a legend and scale, a plant cell with labels, a line chart without values (read from the grid), a circle with a tangent. Each in a clean version and as a crop from a photo (rotation, noise, blur, JPEG 70); plus a series logo (the correct answer is "decoration"). 75 facts (labels, numbers, axes, markings, relations); 18 descriptions per model.
  • No Google, Anthropic or Kimi models in this measurement. All models answered themselves (no redirects).
10 rows · 7 columnsCSV
Qwen/Qwen3.5-397B-A17B98.2%100%96%16Y23.8
Qwen/Qwen3.6-27B97.6%99%96%14Y146.6
ByteDance/Seed-2.0-pro96.3%95%98%13N7.0
Qwen/Qwen3-VL-235B-A22B-Instruct95.7%99%93%14Y15.2
thinkingmachines/Inkling95.1%96%94%11N5.2
XiaomiMiMo/MiMo-V2.6-Pro95.1%98%93%13Y8.5
meta-models/Muse-Glimmer-30B94.5%96%93%11Y33.3
MiniMaxAI/MiniMax-M393.9%94%94%13Y12.6
inclusionAI/Ling-3.0-flash-VL93.3%95%91%12N10.1
deepseek-ai/DeepSeek-V4-Flash-Vision-Exp92.7%95%90%12N4.9

The only gaps of the best model: values from the line chart on the rotated photo (rotation shifts the points relative to the grid by about 2 °C – all models get this wrong). Qwen3.5-397B measured on a longer set of 16 descriptions: facts 143/144 (99.3%) in both runs.