benchmarks.site

Vision & OCR

OCR: exam-sheet photos with handwritten answers

Can models read photos of exam sheets: circled answers, handwritten words, and tell an empty page from a filled one?

Key finding

Gemini 3.1 Pro read circled answers 80/80 and did not invent answers on blank pages (12/12), but misread a handwritten word toward the printed word on the same page depending on the prompt; Qwen3-VL-235B invented answers on a blank sheet during full-page reading yet answered a single yes/no "is there handwriting" question correctly. Samples are very small.

How to read the numbers

Mostly 'correct readings out of attempts', e.g. 18/18. Higher is better.

Hover a column name for its meaning, or see the glossary.

How we tested & caveats · Markdown · CSV

All rows are individual models on a specific task with a photo: reading handwriting, circled answers, recognizing a blank page, transcribing print. The photos are pages of Polish school exam sheets. Samples are very small (one to a few photos, a few to a few dozen repetitions). There is no single "reading accuracy: X%" figure per model here, because nobody transcribed by hand and checked answers across many pages. Costs are the amounts reported by the provider itself in the estimated_cost field of the response (converted to grosze; 1 gr = 0.01 PLN), not calculations from the token price list. Metric definitions and caveats: overview.

1. Reading a single handwritten word (Gemini 3.1 Pro)

  • Model: google/gemini-3.1-pro via DeepInfra (forwarded to Google), reasoning at the medium level (according to the configuration from that period; reconstruction).
  • Data: one photo of a Polish-language exam-sheet page (2026), 1500×2000 px, with a short handwritten open answer. The word under study: the student wrote „opracował” ("worked out / developed"), while the printed text on the same page has „opanował” ("mastered"). The reference was checked by inspecting an enlarged crop of the photo (the first version of the reference was wrong and gave „opanował”).
  • "Prompt variant" means differences in the instruction and in the task description passed along with the photo; the prompt texts are not given here.
4 rows · 5 columnsCSV
prompt variant 1up to 2026-09-231818/180
prompt variant 1, whole-page reading2026-10-1320 readings (10 runs of 2)19/201/20
prompt variant 2 (different description of the tasks on the page)2026-10-1310 readings (5 runs of 2)5/105/10
prompt variant 3 (with a request to correct spelling only)2026-10-1330/33/3

Observations:

  • The error always went toward the word printed on the same sheet. The model "fits" the handwriting to the text it sees in print.
  • In variant 3 the model, in 1 of 3 attempts, also changed a word form („tą” → „tę”, a grammatical form change), even though the prompt asked to correct spelling only. In approx. 80 other readings of this page such a change did not appear.
  • Aggregated (2026-10-13, 40 readings of this page in variant 1): the whole open answer (one sentence) matched the reference word for word in 33/40, and with word agreement ≥ 90% in 40/40. The figures were provided by an agent analyzing the saved readings; they may partly overlap with row 2 of the table.

2. Circled closed-ended answers (Gemini 3.1 Pro)

  • Model as in section 1, the same photo: two closed-ended tasks with a handwritten circle around the answer letter.
  • 2026-10-13: 80/80 correct letters (40 readings × 2 tasks). Each time the model described the mark the same way ("handwritten circle around the letter …") with confidence "high".

3. Blank exam-sheet page

4 rows · 5 columnsCSV
google/gemini-3.1-pro (full page reading)4 photos of exam-sheet pages with no handwriting at all, 1500×2000 px2026-10-1312 (3 per page; the agent reported 15 in total)12/12 with no invented answers
Qwen/Qwen3-VL-235B-A22B-Instruct (full page reading; model reconstructed)rendered blank mathematics exam sheet, 2 pages2026-09-042 runs10 invented answers on 2 pages, described as confidently as real ones; the second run repeated the same error
Qwen/Qwen3-VL-235B-A22B-Instruct (closed question: is there handwriting on the page)the same blank sheet and the same pages with marks drawn in2026-09-04several (exact number unknown)correct: "no handwriting" on blank pages, "handwriting present" on pages with marks
Qwen/Qwen3-VL-235B-A22B-Instruct (the same closed question)photo from section 1 (page with handwriting)2026-10-134040/40 "handwriting present"

Observations:

  • The same model that invented marks on a blank sheet during full-page reading answered correctly when asked about a single thing (whether there is handwriting).
  • Models often claimed to see a "blue circle" on a sheet printed in black and white (qualitative observation from 2026-09-04, no attempt count).

4. Identifying the exam sheet from photos

  • Data: real photos of Polish-language 2026 exam-sheet pages; the exam-sheet code is usually not visible in the photo (it is in the footer).
  • Approx. 2026-09-22.
2 rows · 4 columnsCSV
give the exam-sheet code visible in the photosQwen/Qwen3-VL-235B-A22B-Instruct, google/gemini-3.1-pro (combined)60/6 correct, including 2 times an English-language exam sheet was indicated. Qwen repeated the example values given in the prompt, Gemini indicated an exam sheet from a different year
transcribe a few sentences of print from each of 4 photosboth models7 photo setsprint transcribed usefully in all 7 sets (7/7; no CER measurement)

Observations:

  • A model asked for a code it cannot see gives something plausible-looking instead of "I don't know".
  • A code that was read can differ by one digit ("2505" read as "2605"); qualitative observation.
  • Transcribing whole dense pages of Polish-language print took over 3 minutes and ended in a timeout (the model is not stated in the source, most likely Qwen3-VL-235B).

5. Cost and time of a single call

Costs from the provider's (DeepInfra) estimated_cost field, converted to grosze (gr; 1 gr = 0.01 PLN). Times from sending to the full response, no streaming, from Poland.

8 rows · 6 columnsCSV
google/gemini-3.1-pro (reasoning medium)full reading of a page with handwriting2026-09-175 photos of one exam sheetapprox. 12.3 grn/a
google/gemini-3.1-profull reading of a blank page2026-09-17same as aboveapprox. 3.2 grn/a
google/gemini-3.1-proaverage per page (mix of blank and written pages)2026-09-17same as aboveapprox. 6.9 grn/a
google/gemini-3.1-profull reading of a blank page (1500×2000 px)2026-10-1312n/a6.3–19.7 s (typically 15–20 s)
google/gemini-3.1-profull reading of a page with handwriting (1500×2000 px)2026-10-1313n/aapprox. 25–30 s (indirect estimate, a single call was not measured separately)
Qwen/Qwen3-VL-235B-A22B-Instructclosed question about the page (one-word answer)2026-10n/aapprox. 0.1 gra few seconds (estimate)
Qwen/Qwen3-VL-235B-A22B-Instructtranscribing fragments of print from 4 photos2026-09-227 setsapprox. 0.50 grn/a
google/gemini-3.1-protranscribing fragments of print from 4 photos2026-09-227 setsapprox. 4.1 grn/a

Note: for Gemini, calculating the cost from the token price list and the completion_tokens field underestimated the cost of a page reading approx. 13-fold, because reasoning tokens are billed but not included in completion_tokens (see Platforms).

6. Qwen3-VL-235B-A22B "Thinking" on DeepInfra (2026-09-03)

A request to the Qwen3-VL-235B-A22B-Thinking variant was served by the Instruct model: 0 reasoning tokens in the response, at a rate approx. 2× higher than for Instruct. Details in Platforms. An earlier estimate from the price list (before the measurement) assumed an approx. 6× higher cost for the Thinking variant.

7. Google Document AI (Enterprise Document OCR)

No accuracy measurements in the sources. One early comparison with Gemini 3.1 Pro on one page was scored against the wrong reference (see section 1), so it settles nothing. Qualitative observation: the OCR transcribes everything in a given area, including the printed text of the exam sheet, so its reading of the area around an answer can be several times longer than the student's answer itself. Billing is per page (according to Google's documentation, not measured).