# benchmarks.site > Independent AI model and platform benchmarks on real tasks (OCR of handwriting and documents, Polish speech recognition, diarization, language models, reasoning effort, embeddings, platform behavior, macOS malware detection), plus a curated directory of public AI benchmarks. Every result states date, platform and sample size. Read the methodology and caveats before citing. Cite as: benchmarks.site, with the page URL and measurement date. --- # Methodology & caveats Every result on this site is "model (or engine) X on task Y", with the date, platform, sample size and conditions stated next to it. The pages describe the measurements only, not how any product uses them. ## Speech, vision and text tests Applies to: Polish ASR, diarization, omni (video/audio) models, OCR of handwritten notes, boards and textbooks, language-model classification / summaries / chat, Polish embeddings, and platform availability. Measured September–November 2026. ### Metrics - **WER** (word error rate): the share of erroneous words (substitutions + deletions + insertions) relative to the source text, after normalizing numbers and formulas read aloud (e.g. "1410" = „tysiąc czterysta dziesiąty”, "one thousand four hundred tenth"). Lower = better. Values above 100% are possible (loops, insertions). - **CER** (character error rate): the same at the character level; for photos, computed on canonical text (flattened LaTeX, no formatting and no spaces). - **Facts**: a list of regular expressions for things that must survive (dates, numbers, names, formulas, exercises). E.g. "105/109" = 105 of 109 facts found. For summaries: items of a knowledge list, scored by a judge model. - **DER** (diarization error rate): the share of time with a wrongly assigned speaker, on 10 ms frames. - **Accuracy** (embeddings, classification): the share of correct results on the test; for embeddings, additionally the "margin" = the lowest score of a relevant note minus the highest score of an irrelevant one. - **Latency**: time from sending to the first token / to the end of the response, measured from Poland; depends on the platform's load at that moment. ### Test data - ASR recordings: mostly **speech synthesis** (macOS `say`, Polish and foreign voices), sometimes deliberately degraded (reverb, noise, distance, heavy compression, classroom murmur). Levels: `clean`, `classroom` (phone on a desk in the front rows), `back` (phone at the back of a classroom, strong reverb). **No measurements on real classroom recordings.** - OCR photos: **generated from handwriting fonts** (notebooks, boards, worksheets, textbook pages), with simulated rotation, shadow, noise and compression. Real notebooks will be harder. - Summaries: four synthetic high-school lessons (mathematics, history, biology, Polish) with noise in the style of a speech-recognition transcript. - No real recordings or private content of any person. ### Platforms Most measurements via **DeepInfra** (OpenAI-compatible API, open models and some partner models); additionally **Soniox** (speech recognition, asynchronous and live) and **OpenAI** (transcription models, live mode). Google models (Gemini) on DeepInfra are forwarded to Google. The platform is given with every row. ### Dates 2026-09-22 (ASR, OCR, LLM, embeddings), 2026-09-23 (video/audio), 2026-09-29/30 (ASR and OCR on degraded data, languages, diarization, summaries), 2026-10-04…08 (photos, textbooks, illustrations), 2026-11-05…06 (live speech, provider ASR, chat). No measurements of speech synthesis (TTS) quality or voice-conversation models. ### Caveats 1. **Small samples.** Most rows are one to a dozen or so trials, one or two runs. Differences of 1–3 percentage points are within noise. The sample is given with each result. 2. **Synthetic data** inflates results compared with a real classroom (several voices, people speaking at the same time), although the `back` level is deliberately hard. 3. **Non-deterministic models** (especially Whisper) give different results for the same file; where this was measured, two runs are given. 4. **Platforms change**: queues, limits and model redirects occurred during the measurements. Results reflect the state on the given date. 5. **Judge model**: summaries were scored by a language model; it was sometimes too strict and raised false alarms. 6. "Best in test" applies to the specific task and set, not to the model's overall quality. 7. Own measurements, not independently audited. 8. Processing pipelines, model combinations and costs are deliberately out of scope here. ## Exam-sheet OCR, reasoning and text tests Applies to: OCR of exam-sheet photos, reasoning effort, language models (short answers, grading text, time and cost), and platform errors / billing / prices. Measured September–October 2026. ### Metrics - **Reading accuracy**: readings matching the reference out of attempts, e.g. "18/18". The reference was transcribed by hand from an enlarged photo. For sentences: word-for-word match, or the share of words in common (≥ 90%). - **Cost**: the amount reported by the provider in the response (`estimated_cost`, in USD), converted to grosze (gr), not calculated from the token price list. This matters for models that bill reasoning tokens outside `completion_tokens`. "gr" is the grosz: 1 gr = 0.01 PLN. - **Time**: from sending the request to receiving the full response (no streaming), from Poland. Depends on the platform's load at that moment. - **Reasoning**: a model "reasons" when the response reports a non-zero number of reasoning tokens. - Numbers that were not measured directly but reconstructed (from the total time of several calls, from change history) are marked **reconstruction** or **estimate**. ### Test data - Photos: a few photos of Polish school exam-sheet pages (Polish language 2026, 1500×2000 px): one page with handwritten answers (two circled letters and one sentence of an open answer), four pages with no handwriting; in addition, a rendered blank mathematics exam sheet (2 pages) and sets of photos for identifying the exam sheet. **No measurements on many real handwritten papers.** - Texts: short test queries, steps of a conversation with tools on a test account, answers to exam tasks. - No personal data; handwritten content is not quoted apart from the single word under study. ### Platforms Mostly **DeepInfra** (OpenAI-compatible API; Google Gemini and Anthropic Claude models are forwarded to Google and Anthropic). Some via **OpenAI** (gpt-5, gpt-5.6). Google **Document AI** was used briefly, with no accuracy measurements. The platform is given on every row. ### Dates 2026-09-03 (reasoning, model costs, Qwen3-VL variants), 2026-09-04 (blank page, gemma-4-E4B-it), 2026-09-17 (cost of a page reading with Gemini 3.1 Pro), approx. 2026-09-22/23 (exam-sheet identification, handwritten word 18/18), 2026-10-13 (reading handwriting and blank pages, DeepSeek-V4-Flash with and without a date, error 402), 2026-10-16 (DeepInfra model list and prices). No quality measurements for Claude Opus 5.5, Kimi, GLM, gpt-oss, or for the accuracy of answer grading. ### Caveats 1. **Very small samples.** Most rows are one photo and a few to a few dozen repetitions. A result applies to the specific photo, not to the model's general quality. 2. **No test of grading accuracy.** Nobody compared model grades with an examiner's grades on many papers. 3. **Some numbers are reconstructed** from total times, saved readings and change history; this is marked on the row. 4. **Models are non-deterministic**: the same photo produced different readings; results also depend on the wording of the prompt. 5. **Platforms change**: overloads, redirects between model variants and lack of account balance occurred during the measurements. Results reflect the state on the stated date. 6. Own measurements, not independently audited. 7. Processing pipelines, prompts and configuration are deliberately out of scope here. ## macOS malware-scanning tests Applies to: the malware-detection pages, static-heuristic false alarms, ML classifiers on TLSH fingerprints, and scanner performance. Measured November 2026 on a macOS malware scanner under development. Only aggregates are published: counts, rates, per-category numbers, names of public test files and malware families, public SHA-256 hashes, and detection engine / signature names. ### Verdict levels and thresholds Each file gets one of the verdicts `CLEAN`, `INFO`, `SUSPICIOUS`, `PUA` or `MALICIOUS`, based on a score summed from individual detections (heuristics, YARA rules, ClamAV / XProtect signatures, hash databases, fuzzy TLSH matching). Results are reported at three cumulative thresholds: | Threshold | Counts as a hit | Meaning | |---|---|---| | INFO+ | `INFO`, `SUSPICIOUS`, `PUA`, `MALICIOUS` | Anything reported at all, including low-weight informational findings that are not shown as an alarm. | | SUSPICIOUS+ | `SUSPICIOUS`, `PUA`, `MALICIOUS` | What a user would see as an alarm. This is the threshold used for the false-alarm target. | | MALICIOUS | `MALICIOUS` | High-confidence detection only. | In the held-out benchmark the counts are consistent with this: 318 `MALICIOUS` + 141 `SUSPICIOUS` + 20 `PUA` = 479 at SUSPICIOUS+, and 479 + 40 `INFO` = 519 at INFO+ (out of 528 test samples). ### Test data - **Malicious test samples:** 528 files in seven groups — `yara/scripts_web` 312, `yara/gen` 107, `yara/macos` 60, `yara/native` 16, `scripts` 16, `documents` 15, `eicar` 2 — plus 14 clean test samples and 48 neutral archive files (no expected verdict) in the static-heuristics fixtures. - **Clean files from a real Mac:** installed applications, Homebrew packages, Python libraries, application support data, installers, documents and dependencies. Only category names and counts are published. - **Held-out clean set:** clean files are split into a set used while tuning heuristics (2309 files; 2077 in the 2026-11-09 run) and a held-out set never used for tuning (2973 files; 2812 in the 2026-11-09 run). The held-out false-alarm rate is the honest number; the tuning-set rate is shown only for comparison. - **Safe benchmark:** only public, harmless test files (eicar.org, AMTSO Security Features Check) and public data (SHA-256 hashes from Objective-See reports, family names). ### Caveats - Sample sizes are small for some categories (e.g. `project_deps` 4 files, `user_documents` 22, `installers_archives` 30, clean test samples 14); a single file changes the rate by several percent. - Zero false alarms at SUSPICIOUS+ on 2973 held-out files does not mean a zero rate: by the rule of three the 95% upper bound is about 3/2973 = 0.10%. - 495 of the 528 malicious test samples are in the `yara/*` groups (test fixtures for the YARA rule set), so the detection rate is not a measure against unknown, fresh malware. The Objective-See hash test (8/33) and the family coverage test (57/83) show the gaps on real-world Mac malware more directly. - The hash and family tests check database / rule coverage, not scanning of live samples. - All measurements were taken on a single Mac; timings are wall-clock and were not taken under controlled load. Manual performance measurements have no raw result files. - Not measured yet: local LLMs (Foundation-Sec vs others), databases (SQLite / DuckDB / LMDB / RocksDB), and other antivirus engines (ClamAV `clamscan`) on the safe benchmark. --- # ASR: Polish speech recognition > On clean synthetic speech, Qwen3-ASR-1.7B and Gemini 3.1 flash-lite kept Polish–English mixing verbatim while Whisper-large-v3 translated it; on degraded recordings the external providers Soniox stt-async-v6 and OpenAI gpt-transcribe had far lower WER (8.9% and 14.0% at the back-of-classroom level vs 34.4–63.8% for the DeepInfra models), and Soniox stt-rt-v6 had the lowest WER in live streaming. All test audio is synthetic. Area: Speech. Page: https://benchmarks.site/asr-polish/. Methodology and caveats: https://benchmarks.site/methodology/#speech-vision-text # Speech recognition (ASR) in Polish All rows are results of **single models**, without text correction by a second model and without merging transcripts. Metrics: WER (lower = better), facts (how many of 109 fact expressions were found in the transcript), handling of Polish–English mixing. Metric definitions and caveats: [overview](/methodology/#speech-vision-text). ## 1. Test of 2026-09-22: clean speech synthesis, Polish + English in one recording - Platform: DeepInfra (Gemini: via DeepInfra to Google). - Data: macOS synthetic speech (Zosia pl_PL, Samantha en_US), 16 kHz mono. Clips: `pl` 48.5 s math lecture with formulas and numbers; `en` 29.4 s biology in English; `mix` 20.1 s Polish-English-Polish; `mixEN1` 20.1 s English first; `noisy` = the `mix` clip + pink noise and 100 Hz hum, approx. 10 dB SNR (mp3 32 kb/s and Opus); `long` 45.85 min (repeated clips with silence). - 178 calls in total. WER is "approximate" (light normalization of numbers and formulas). In the mixed clip, a translated or omitted English sentence counts as approx. 45% WER. - A test on **clean speech synthesis** says nothing about a real classroom (see section 2). | model | platform | WER PL clean | WER EN | mix clean, no language hint | mix noisy | runs with English preserved (verbatim) | latency, 20–48 s clip | 45-min file in one request | |---|---|---|---|---|---|---|---|---| | Qwen/Qwen3-ASR-1.7B | DeepInfra | 2.5% | 0% | 6.4% (verbatim) | 12.8% | 17/17 | 1.8–9 s | 15–27 s, 42/42 English sentences preserved | | Qwen/Qwen3-ASR-0.6B | DeepInfra | 18.0% | 0% | 8.5% | 21–23% | 12/12 | 1.6–5.5 s | not tested | | openai/whisper-large-v3-turbo | DeepInfra | 4.9% (drops words at the 30 s window boundary) | 0% | 2.1% when correct, 45–55% when hallucinating | 2–4% | 10/22 | 1.5–3.9 s (up to 10 s) | 13.5 s, only 21/42 English sentences preserved; 88 MB WAV file accepted | | openai/whisper-large-v3 | DeepInfra | 6.6% | 0% | 44.7% (English sentence translated into Polish every time) | 46.8% (English omitted) | 0/12 | 5.5–23 s | not tested | | mistralai/Voxtral-Small-24B-2507 | DeepInfra | 77% without a language hint (translates into English); 4.9% with Polish specified, but wrong month („maja” ("of May")) | 0% | 44.7% (English omitted) | 2.1% or 44.7% (inconsistent) | 0/10 | 3–8.8 s | not tested | | mistralai/Voxtral-Mini-3B-2507 | DeepInfra | 64.8% without a language hint (output in Russian); 9% with a hint (output in LaTeX) | 1.2% | 76.6% (everything in English) | 94–96% | 0/10 | 1.5–3.3 s | not tested | | nvidia/Nemotron-3.5-ASR-Streaming-Multilingual-0.6b | DeepInfra | 47.5% (40% with Polish specified) | 0% | 23.4% (writes in Cyrillic) | 38% | 6/12 (only with the language specified) | 1.6–5.6 s | not tested | | google/gemini-3.1-flash-lite (chat model with audio input) | DeepInfra → Google | 0–1% | 0% | 0% (verbatim) | 0% | 3/3 | 2.5–3.4 s; 10 min in 10.5 s | 46 min without streaming: 300 s timeout exceeded | | google/gemini-3.5-flash (chat model with audio input) | DeepInfra → Google | approx. 1% | n/a | n/a | 0% | 1/1 | 2.7–3.6 s | n/a | Noisy mixed clip: Qwen3-ASR-1.7B has 12.8%, Gemini 3.1 flash-lite 2.1%; example of a Qwen error: „krótki fragment tekstu” ("a short piece of text") → „trubki Pradmentowskiego” (nonsense). Observations (samples as above): - Whisper-large-v3 picks one language per file and translates the rest: 12 of 12 mixed runs (English sentence translated into Polish); when English came first, it translated the Polish. - Whisper-large-v3-turbo is non-deterministic even at temperature 0: in 11 of 22 runs it replaced the English sentence with a hallucination („Wszystkie prawa zastrzeżone” ("All rights reserved"), „Współpraca z Wrocławiem” ("Cooperation with Wrocław"), loops of „w tym roku” ("this year")). Both Whispers drop words at the 30 s window boundary („Delta wynosi dwadzieścia pięć” ("Delta equals twenty-five") → „5-24”). An additional text prompt for turbo emptied the output on the noisy clip. - Qwen3-ASR gives identical output on repeats and returns word timestamps from forced alignment (English sentence started at 7.44 s vs the true 7.40 s), but has coarse segments (one per 20–27 s) and reports only the dominant language. - Voxtral returns no timestamps or language; the webm and m4a formats end in an HTTP 500 error; the model is not available through the chat interface (404). - Gemini: only mp3 and wav (ogg: error 422); no word timestamps; the 1024-token output limit truncated a 10-minute chunk. - Formats: mp3, ogg/opus, wav, flac accepted by all 7 ASR models; webm/opus and m4a/aac by Whisper, Qwen and Nemotron. ## 2. Degraded recordings, 2026-09-29 (single models, no text correction) - Platform: DeepInfra. - Data: 8 topics (mathematics, physics, history, chemistry, biology, English PL+EN, Polish, civics (WOS)), approx. 9 minutes in total, speech synthesis, 109 facts (dates, numbers, names, formulas). Formulas and numbers read aloud. Three levels: `clean` (microphone at the mouth), `classroom` (phone on a desk in the front rows: reverb RT60 0.6 s, SNR 20 dB, AAC compression 64 kb/s), `back` (phone at the back of the classroom: reverb 0.85 s, SNR 14 dB, classroom murmur, AAC compression 48 kb/s). - One run per cell unless noted otherwise. Depending on the test variant, Qwen's result at the `back` level fell within 58.5–64.1%, and Whisper v3's within 31.5–47.4%. | model | platform | clean: WER (facts/109) | classroom: WER (facts/109) | back: WER (facts/109) | notes | |---|---|---|---|---|---| | Qwen/Qwen3-ASR-1.7B | DeepInfra | 10.9% (80) | 25.8% (61) | 63.8% (20) | deterministic; keeps English verbatim; in reverb loses about two thirds of the words | | Qwen/Qwen3-ASR-0.6B | DeepInfra | 26.8% (60) | n/a | 88.1% (0) | unfit for Polish speech in reverb | | openai/whisper-large-v3 | DeepInfra | 13.3% (89)\* | 15.8% (82)\* | 34.4% (51) in one run, 47.4% (50) in the second | \*one 30-s window lost; non-deterministic; translates English sentences into Polish; once every few runs turns a whole window into gibberish or drops it (topic "history" 55–71% WER even on clean) | | openai/whisper-large-v3-turbo | DeepInfra | n/a | n/a | 41.9% (47) | one run | | MiMo-V2.6-Pro and MiMo-V2.6-Flash (chat model, audio in the message) | DeepInfra | n/a | n/a | > 800% | loops („w tym samym roku…” ("in the same year…")) | | Inkling, Inkling-Small, gemma-4-31B-it | DeepInfra | n/a | n/a | n/a | the interface rejects audio input (HTTP 400) | Additionally: longer chunks for Whisper (60 s instead of ≤ 30 s) gave 45.8% WER at the `back` level (more dropped words). The DeepInfra model list as of 2026-09-29 contained no new ASR models beyond Whisper, Qwen3-ASR, Voxtral and Nemotron. ### Earlier measurement, 2026-09-23 (one deliberately degraded recording) A recording imitating a lesson heard from a desk: reverb, pink noise, distance, 48 kb/s compression. WER against the source text, 1 recording, no text correction. The source does not state precisely whether the audio was preprocessed before models other than Qwen. | model | platform | WER PL | WER PL+mix (average) | English preserved | notes | |---|---|---|---|---|---| | Qwen/Qwen3-ASR-1.7B, raw audio | DeepInfra | 19.0% | 26.0% | 71% | | | openai/whisper-large-v3-turbo | DeepInfra | 23.3% | n/a | n/a | | | openai/whisper-large-v3 | DeepInfra | 24.8% | n/a | n/a | | | Voxtral Mini and Small | DeepInfra | ≥ 90% | n/a | n/a | unfit for Polish | | Nemotron-Diarization (used as ASR) | DeepInfra | ≥ 90% | n/a | n/a | a speaker-recognition model, not a transcription model | ## 3. Language lessons: Polish and a foreign language in one sentence, 2026-09-29/30 - Platform: DeepInfra. - Data: 4 lessons, 7.6 min in total: Spanish ×2 (pretérito indefinido; „en el restaurante”), English (present perfect), German (Perfekt). The Polish parts are read by synthetic Zosia, the foreign parts by synthetic native voices, so a sentence switches language in the middle. Averages over runs (Whisper run twice). Small sample: 4 lessons. - "Preserved" = the foreign word written as spoken; "translated" = written in Polish; "garbled" = in the original language with errors; "lost" = missing, gibberish or phonetic spelling in Polish letters. Non-preserved fragments were scored by a judge model. All languages together, single models: | model | level | WER overall | WER of Polish fragments | foreign preserved | translated | garbled | lost | |---|---|---|---|---|---|---|---| | Qwen3-ASR-1.7B | clean | 15.4% | 16.0% | 86.1% | 0.5% | 5.8% | 7.7% | | whisper-large-v3 | clean | 34.3% | 10.8% | 41.6% | 30.7% | 7.9% | 19.8% | | Qwen3-ASR-1.7B | classroom | 35.6% | 31.7% | 62.1% | 10.9% | 10.9% | 16.1% | | whisper-large-v3 | classroom | 33.4% | 29.9% | 68.4% | 19.3% | 8.4% | 4.0% | | Qwen3-ASR-1.7B | back | 65.5% | 67.6% | 40.5% | 7.7% | 13.4% | 38.4% | | whisper-large-v3 | back | 56.5% | 60.1% | 56.1% | 7.8% | 7.3% | 28.8% | Whisper-large-v3 by foreign language (classroom / back): | language | WER classroom | foreign preserved / translated, classroom | WER back | foreign preserved / translated, back | |---|---|---|---|---| | Spanish | 39.1% | 60.1% / 27.2% | 63.9% | 44.3% / 4.5% | | English | 14.5% | 90.3% / 2.6% | 32.1% | 85.1% / 0.0% | | German | 41.7% | 61.2% / 21.4% | 67.3% | 48.5% / 23.3% | Observations: Qwen3-ASR keeps both languages, but in noise it writes foreign words phonetically in Polish („a djeł, lasy na paszare” instead of „ayer, la semana pasada”). Whisper picks one language in every 30 s window and translates or drops the other (a Spanish reading passage came out as „Wczoraj wyszłam do kina z moimi przyjaciółmi” ("Yesterday I went to the cinema with my friends"), in Polish). German at the back of the room was unreadable for Whisper (WER approx. 67%). ## 4. External providers on the same recordings, 2026-11-06 - Platforms: Soniox (`stt-async-v6`, file mode), OpenAI (transcription models). - Data: the same 9 minutes, 8 topics, 109 facts, three levels (clean / classroom / back); plus 79 numbers and 54 proper names. Speech synthesis. Single model without text correction; the number of runs for these rows is not given in the source. - WER and count columns: clean / classroom / back. | model | platform | WER % | facts out of 109 | numbers out of 79 | names out of 54 | compute time, s per minute of recording | word timestamps | |---|---|---|---|---|---|---|---| | Soniox stt-async-v6 | Soniox | 2.4 / 3.4 / 8.9 | 103 / 97 / 87 | 77 / 77 / 77 | 52 / 50 / 42 | 8.6–14.8 | yes | | gpt-transcribe | OpenAI | 1.2 / 4.7 / 14.0 | 107 / 104 / 92 | 78 / 77 / 75 | 54 / 53 / 43 | 2.8–4.3 | no | | gpt-4o-transcribe | OpenAI | 16.7 / 16.8 / 30.7 | 79 / 76 / 65 | 67 / 67 / 67 | 27 / 29 / 19 | 3.5–5.6 | no | | gpt-4o-mini-transcribe | OpenAI | 16.4 / 19.8 / 61.5 | 82 / 82 / 51 | 68 / 70 / 55 | 29 / 39 / 15 | 2.3–2.7 | no | | gpt-4o-transcribe-diarize (classroom level only) | OpenAI | 11.2 | 88 | 77 | 44 | 29.9 | segments | Notes: the `gpt-4o-*` models were measured with the language specified; without it, at the `classroom` level they did no better (18.6% and 20.3%). OpenAI has announced the shutdown of the `gpt-4o-*` models on 2027-02-26. Language lessons (mixed sentences) were not measured with these providers. Processing time for a 45.6-min lesson (the classroom-level recording repeated 5 times, one measurement): Soniox in file mode, 78 s. ## 5. Live speech (streaming), 2026-11-05/06 - Platforms: Soniox (WebSocket), OpenAI (live mode), DeepInfra (Nemotron and Qwen3-ASR in 20 s chunks). - Data: 3 lessons from fixtures (history / mathematics / biology), speech synthesis; the text contains fixture markers that are read aloud, which creates a floor of approx. 6–10% WER for all models, so this set barely shows differences between the good models. Nemotron and Qwen received separate 20 s chunks, the others a continuous stream (incomplete comparison). | model | platform | WER clean (hist./math/biol.) | WER classroom (hist./math/biol.) | |---|---|---|---| | Nemotron 3.5 ASR Multilingual, language specified (20 s chunks) | DeepInfra | 26.5 / 29.5 / 31.4% | 43.7 / 39.9 / 46.3% | | Qwen3-ASR (20 s chunks) | DeepInfra | 17.2 / 11.3 / 69.9%\* | 28.5 / 16.4 / 24.2% | | Soniox stt-rt-v3 | Soniox | 9.3 / 6.4 / 6.9% | 10.7 / 6.4 / 8.2% | | Soniox stt-rt-v6 | Soniox | 8.2 / 6.4 / 6.6% | 7.6 / 6.2 / 7.4% | | OpenAI gpt-live-transcribe | OpenAI | 9.0 / 10.2 / 10.9% | 12.7 / 9.4 / 12.8% | | OpenAI gpt-realtime-whisper | OpenAI | 9.3 / 10.5 / 10.4% | 11.5 / 10.2 / 9.8% | \* Outlier run of Qwen; the other runs were 11–28%. Nemotron without the language specified: 37–45% WER and output in Cyrillic. Latency and additional measures (link with no other traffic, at most 2–3 streams at once): | model | first partial result | end of utterance → result (min / median / max) | WER at pauses | numbers / names from the lesson | |---|---|---|---|---| | Soniox stt-rt-v3, PCM | 961–1009 ms | 280 / 499 / 1709 ms | 4.0% | 168/202, 65/68 | | Soniox stt-rt-v6, PCM | 1254–1316 ms | 254 / 461 / 691 ms | 3.1% | 167/202, 64/68 | | OpenAI gpt-live-transcribe | 861–3296 ms under load; 1268–1337 ms at pauses | 185 / 484 / 1643 ms (last text) | 5.0% | 141/202, 68/68 | | OpenAI gpt-realtime-whisper | 1503–7157 ms under load; 1673–1769 ms at pauses | 598 / 837 / 1310 ms (last text) | 6.6% | 138/202, 68/68 | - Soniox provides an end-of-utterance marker and word timestamps; OpenAI in live mode did not provide them. - Language hint (Soniox, average WER across lessons): `stt-rt-v3` with a list of languages 8.0%, `pl` only 8.5%, no hint 9.6%; `stt-rt-v6` with a list 7.1%, `pl` only 7.4%. Differences are approximate (some runs went over a loaded link). - Foreign words pronounced the Polish way inside a Polish sentence (Soniox `stt-rt-v6`, 2026-11-10, 6 synthetic recordings) came out wrong, e.g. "altog" instead of "although". Adding a list of difficult words to the context changed the result from 3/6 to 6/6 correct recordings (very small sample). - Nemotron on DeepInfra returns no partial results (despite "Streaming" in its name): a 3–6 s window takes 1.0–1.9 s of recognition alone (median 1.3 s). A 20 s chunk: 1.4–3.5 s (Nemotron) vs 50–77 s (Qwen3-ASR, under DeepInfra load on 2026-11-05); 10 chunks at once: 7–22 s vs 61–148 s. --- # Diarization: who is speaking > On four short synthetic lessons, nvidia/Nemotron-3-Diarization (DeepInfra) averaged 9.4% DER and detected 31 of 36 overlapping-speech events, while Soniox stt-async-v6 and OpenAI gpt-4o-transcribe-diarize detected none; note that a trivial "one speaker" answer already scores 13–18% DER on this set. Area: Speech. Page: https://benchmarks.site/diarization/. Methodology and caveats: https://benchmarks.site/methodology/#speech-vision-text # Diarization: recognizing who is speaking Metric: **DER** (diarization error rate; lower = better), computed on 10 ms frames, including speech by several people at once, consistent with pyannote.metrics to within 0.1–0.2 points. Reference point: the trivial answer "all speech belongs to one speaker" gives a DER of 13–18% on this set (the teacher speaks 82–87% of the time), so a DER of 6–14% looks good mainly because one speaker dominates. Metric definitions and caveats: [overview](/methodology/#speech-vision-text). ## 1. nvidia/Nemotron-3-Diarization (DeepInfra), 2026-09-30 - The model returns only speaker turns (anonymous labels `speaker_N`, up to 8 speakers, consistent within a file, 10 ms resolution, no confidence and no text); turns of different speakers overlap when several people speak at once. The output is deterministic (the same on every call on the same file). - Data: 4 synthetic lessons, 7.1 min in total: history, biology, mathematics, English; 3–4 students in each, 4–5 places with two people speaking at once. In three lessons the students are a pitch-shifted (higher) synthetic Zosia, so the voices differ mainly in pitch (a hard case); the English lesson has clearly different voices. The `classroom` and `back` levels are as in [ASR](/asr-polish/) (section 2). **Synthetic speech, no real children.** - "Groups / speakers" = number of detected voice groups / number of true speakers. "Overlap: recall / precision" = frames in which several people speak at once. In parentheses at the `back` level: DER according to the source with a 0.25 s collar; for `classroom` the source gives a single value. | recording | level | DER | groups / speakers | several people at once: recall / precision (frames) | "overlap" events detected | |---|---|---|---|---|---| | history | classroom | 6.8% | 3 / 4 | 94% / 94% | 5/5 | | biology | classroom | 6.2% | 3 / 5 | 98% / 90% | 4/4 | | mathematics | classroom | 11.5% | 3 / 5 | 82% / 88% | 3/4 | | English | classroom | 11.7% | 4 / 5 | 75% / 88% | 4/5 | | history | back | 8.3% (5.5%) | 3 / 4 | 84% / 87% | 4/5 | | biology | back | 6.5% (4.4%) | 3 / 5 | 98% / 88% | 4/4 | | mathematics | back | 13.5% (9.6%) | 2 / 5 | 85% / 92% | 3/4 | | English | back | 11.1% (7.0%) | 4 / 5 | 91% / 87% | 4/5 | Conclusions: - Speech by several people at once is a strong point: 75–98% of frames, 31 of 36 events detected. - Students whose voices differed only in pitch (pitch-shifted TTS) were sometimes merged into one group or attached to the teacher: the model distinguished 0–1 of 3–4 students, and short answers („Czterdzieści dziewięć.” ("Forty-nine.")) went to the teacher's group. With clearly different voices (English) it found 4 groups for 5 people. Results on real children may differ. - Time and scale: a 45-minute lesson (21.9 MB mp3) in one request: 16.7 s, 4 groups, 189 turns. - The model is a streaming model (with speaker memory); the file was not split. ## 2. Comparison with external providers, 2026-11-06 The same 4 lessons, `classroom` and `back` levels, the same DER measure. Results are for single models; WER is not given because it does not apply to diarization itself. | model | platform | DER average | several people at once detected | notes | |---|---|---|---|---| | nvidia/Nemotron-3-Diarization | DeepInfra | 9.4% (classroom 9.1%) | 31 of 36 | | | Soniox `stt-async-v6` (diarization in file mode) | Soniox | 13.6% (classroom 12.7%) | 0 of 36 | at the level of the "one speaker" answer; does not capture speech by several people | | OpenAI `gpt-4o-transcribe-diarize` (classroom only) | OpenAI | 18.1% | 0 of 18 | provides labeled segments; processing time 29.9 s per minute of recording | Limitations: one run per model, small sample (4 short lessons), synthetic speech. --- # Omni models: video and audio input > On DeepInfra, `video_url` input delivers only image frames to the model (audio was dropped for 6 of 6 models), and models see a fixed number of frames (32 or 16 where measured) regardless of video length; audio has to be sent as a separate `input_audio` part (wav/mp3 only, max 600 s per part). Area: Speech. Page: https://benchmarks.site/omni-video-audio/. Methodology and caveats: https://benchmarks.site/methodology/#speech-vision-text # "Omni" models: video and audio as input Platform: DeepInfra. Measured on 2026-09-23 against the live API, supplemented with results from 2026-09-29. All numbers come from API responses (`usage`), not from documentation. Metric definitions and caveats: [overview](/methodology/#speech-vision-text). ## Catalog (2026-09-23) From the DeepInfra model list: 232 models not marked as deprecated, including 17 with a video-input tag, 11 with an audio-input tag and 8 native ASR models. The tag is binding: a model without the tag responds with HTTP 405 ("does not accept video input"); e.g. `google/gemini-3.1-flash-lite` does not accept video (audio only). All Gemini models on DeepInfra are forwarded to Google (partner flag); `google/gemma-4-31B-it-Ultra` is also a partner model (despite open weights), while the plain `gemma-4-31B-it` and `-turbo` are hosted by DeepInfra. ## 1. Does the model hear the audio from a video file? - Test clip: 7.7 s, a slide showing `x + 7 = 12` / `Zadanie domowe: str. 42` ("Homework: p. 42"), a speech track with a "secret word" („bakłażan” ("eggplant")). Prompt: transcribe the board and quote what can be heard; if nothing can be heard, write `BRAK DZWIEKU` ("NO SOUND"). 1 clip, 1 trial per model. | model | board read | audio from `video_url` | |---|---|---| | google/gemma-4-31B-it (video + audio) | yes | no audio | | XiaomiMiMo/MiMo-V2.6-Flash (video + audio) | yes | no audio | | XiaomiMiMo/MiMo-V2.6-Pro (video + audio) | yes | no audio | | Qwen/Qwen3.5-35B-A3B | yes | no audio | | inclusionAI/Ling-3.0-flash-VL | yes | no audio | | zai-org/GLM-5.3-Flash | yes | no audio | Conclusion: **on DeepInfra, `video_url` input delivers only image frames to the model; the audio is discarded** (six models out of six). Evidence: the same MiMo-V2.6-Flash transcribed the same utterance word for word when it received it as a separate audio part, and with video and audio in one message it responded to both. ## 2. How many frames does the model see? The number of input tokens for video was **identical** for videos of 30 s, 60 s, 300 s, 720 s and 2700 s, and for 1280×720 and 640×360 – a fixed number of frames regardless of length. No video length limit was found (a 2700 s video went through), nor any way to set the number of frames. | model | tokens per video (any length) | tokens per 1 full image | frames | |---|---|---|---| | google/gemma-4-31B-it | 2421 | 266 | 32 | | inclusionAI/Ling-3.0-flash-VL | approx. 4400 | 883 | approx. 32 | | Qwen/Qwen3.5-35B-A3B | approx. 11720 | 883 | approx. 32 | | XiaomiMiMo/MiMo-V2.6-Flash | approx. 14250 | 882 | 16 | | XiaomiMiMo/MiMo-V2.6-Pro | 14254 | n/a | 16 | | zai-org/GLM-5.3-Flash | 9732 | n/a | n/a | The frame count was checked directly: a 40 s video with a different caption `KOD 01`…`KOD 40` ("CODE 01"…"CODE 40") every second, and the prompt "list the frame numbers you saw". gemma-4-31B-it listed exactly 32 evenly spaced numbers; MiMo-V2.6-Flash 16 numbers, approx. every 2.5 s; **Qwen3.5-35B-A3B listed 39 of 40, although the token count implies approx. 32 frames** – i.e. it fills in frames it did not see (a risk of invented content for a board with numbers). Converted to a 45-minute lesson: one frame per approx. 84 s (gemma/Qwen/Ling, 32 frames) or per approx. 169 s (MiMo, 16 frames). ## 3. Audio as a separate message part - `input_audio` format: only `wav` and `mp3`. Hard limit of 600 s per part (above it, the error "Audio exceeds maximum allowed duration of 600s"). - A 40 MB payload (base64) went through. - Speech quality in omni models (one clip, synthetic, clean): MiMo-V2.6-Flash garbled a rare word („bakłażan” → „baczurzan”); `google/gemini-3.1-flash-lite` transcribed it without error. - On a degraded recording (`back` level, 9 min, synthesis; [ASR](/asr-polish/), section 2): MiMo-V2.6-Pro and -Flash: WER > 800% (loops of „w tym samym roku…” ("in the same year…")); `thinkingmachines/Inkling`, `Inkling-Small` and `google/gemma-4-31B-it` – the interface rejected audio input (HTTP 400) or the model was unfit for Polish transcription. - Omni models returned plain text only, with no word timestamps. --- # OCR: handwritten notes, boards and textbook pages > On synthetic Polish notes, handwriting and textbook pages, Kimi-K2.6, Qwen3.5-397B-A17B and DeepSeek-V4.1-Flash were among the most accurate single models, and on print all tested models stayed below 1.5% CER. Small numbers are the weak spot - on one test-schedule page every model "corrected" 2026 to 2025 - and on tiny or distant writing models reconstruct content instead of reading it. Area: Vision & OCR. Page: https://benchmarks.site/ocr-handwriting-boards-textbooks/. Methodology and caveats: https://benchmarks.site/methodology/#speech-vision-text All rows are **single models** reading one photo. There are no results from combining several readings here. These are general-purpose (vision-language) models; the DeepInfra model list from September 2026 had no dedicated OCR model (olmOCR, dots.ocr, PaddleOCR-VL, DeepSeek-OCR) – see section 3 on redirects. Metrics: CER (lower = better), "diacW" = share of words with Polish diacritics written exactly, "math" = share of passed formula checks (LaTeX), "nums" = share of numbers read correctly (dates, 3.14, 87, 5.18…), facts = share of fact expressions found in the reading. Models with a reasoning mode were measured with reasoning disabled unless noted otherwise. All photos are **synthetic** (handwriting from fonts + simulated degradation); real notebooks will be harder. ## 1. Test from 2026-09-22: 4 photos, 19 models - Platform: DeepInfra (Google models – through DeepInfra to Google; ByteDance Seed – upstream ByteDance/BytePlus). - Data: 4 images. (1) `notes_clean` – Caveat font on squared paper, 1200×1600; (2) `notes_photo` – the same with rotation, perspective, shadow, blur, noise, contrast ×0.7, JPEG q60, 1320×1720; (3) `board_photo` – Kalam font on a whiteboard with strong glare, 2 columns, JPEG q70; (4) `cursive_photo` – Marck Script cursive handwriting, denser, JPEG q60, 1300×1700 (the hardest). Content: Polish diacritics, the pangram „Zażółć gęślą jaźń / źdźbła łódź” (a Polish pangram packed with diacritics), x=(−b±√Δ)/2a, E=mc², H₂SO₄, reactions, limits, derivatives, dates, page numbers. - 251 calls. CER computed on the main text of images 1, 2 and 4. Number of checks: 14 (notes), 9 (whiteboard), 11 (cursive). "date" = whether the small date on the degraded photo was read as 22.09.2026. "seq" = total latency / time to first token, streaming, one request at a time; "conc" = median latency with about 10 parallel requests; "inTok" = input tokens for the 1320×1720 photo (with a short instruction). | model | CER | diacW | math | nums | date | seq (s) | conc (s) | inTok | |---|---|---|---|---|---|---|---|---| | google/gemini-3.1-flash-lite | 0.018 | 1.00 | 1.00 | 0.97 | N (2025) | 4.3 / 2.8 | 4.2 | 1305 | | ByteDance/Seed-2.0-mini | 0.019 | 0.99 | 0.98 | 1.00 | Y | 10.9 / 4.9 | 7.4 | 1515 | | google/gemma-4-31B-it-turbo | 0.019 | 0.99 | 0.98 | 0.89 | N (2020) | 48–112 / 3–7 | 21.6 | 490 | | google/gemma-4-31B-it | 0.019 | 0.99 | 0.98 | 0.89 | N (2023) | n/a | 22.8 | 490 | | Qwen/Qwen3.6-35B-A3B | 0.022 | 0.99 | 1.00 | 0.89 | Y | 80 / 55 | 58.8 | 1169 | | Qwen/Qwen3.5-35B-A3B | 0.025 | 0.96 | 1.00 | 0.89 | Y | 6.4 / 2.1 | 4.3 | 1170 | | deepseek-ai/DeepSeek-V4.1-Flash | 0.026 | 0.97 | 1.00 | 0.89 | N (2025) | 9.5 / 2.6 | 9.8 | 1189 | | inclusionAI/Ling-3.0-flash-VL | 0.028 | 0.93 | 1.00 | 0.89 | N (2025) | 8.7 / 2.2 | 7.9 | 2446 | | Qwen/Qwen3-VL-235B-A22B-Instruct | 0.029 | 0.95 | 1.00 | 0.89 | N (2025) | 32 / 2.7 | 42.7 | 2430 | | XiaomiMiMo/MiMo-V2.6-Flash | 0.033 | 0.93 | 0.98 | 0.87 | skipped | 4.7 / 2.1 | 4.6 | 2430 | | Qwen/Qwen3.8-27B | 0.034 | 0.89 | 0.98 | 0.83 | N | n/a | 26.1 | 2438 | | deepseek-ai/DeepSeek-V4-Flash-Vision-Exp | 0.038 | 0.89 | 0.95 | 0.88 | Y | n/a | 15.7 | 549 | | Qwen/Qwen3-VL-30B-A3B-Instruct | 0.045 | 0.92 | 1.00 | 0.89 | N (2025) | 43 / 2.3 | 22.3 | 2430 | | google/gemma-3-27b-it | 0.047 | 0.92 | 0.87 | 0.81 | Y | 27 / 2.8 | 17.8 | 472 | | meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8 | 0.056 | 0.84 | 0.96 | 0.87 | skipped | 5.9 / 2.1 | 6.9 | 2677 | | meta-llama/Llama-4-Scout-17B-16E-Instruct | 0.063 | 0.82 | 0.93 | 0.82 | skipped | n/a | 7.9 | 2677 | | mistralai/Mistral-Small-3.2-24B-Instruct-2506 | 0.065 | 0.81 | 0.94 | 0.85 | skipped | n/a | 8.5 | 2631 | | google/gemma-3-12b-it | 0.078 | 0.91 | 0.91 | 0.82 | N (22.07) | n/a | 15.1 | 472 | | Qwen/Qwen3.5-9B (reasoning not disabled) | 0.369 | 0.70 | 0.79 | 0.77 | Y | n/a | 47 | 1167 | | google/gemini-3.5-flash (reference, cursive only) | 0.023 | 1.00 | 1.00 | 1.00 | – | n/a | 5.5 | 1305 | | google/gemma-4-26B-A4B-it | 0.023 | 1.00 | 0.91 | 0.83 | – | n/a | 98–180 (2 of 4 timed out) | 490 | | zai-org/GLM-5.3-Flash | – | – | – | – | – | all 4 calls with a photo timed out (180–400 s) | – | – | Observations: - The word „źdźbła” ("blades [of grass]") was missed most often (most models "normalized" it to żdźbła/zdźbła); read correctly by, among others, gemini-3.1-flash-lite. - The small date and small numbers on the degraded photo are unreliable: the year 2026 was read as 2025 (gemini-3.1-flash-lite, Qwen3-VL-235B/30B, DeepSeek-V4.1-Flash, Ling, Qwen3.8), as 2020/2023 (gemma-4), and Llama-4, Mistral and MiMo skipped the date. In the cursive text „5.18” often became „518”. Only these read the date correctly: Seed-2.0-mini, Qwen3.5-35B, Qwen3.6-35B, gemma-3-27b, DeepSeek-V4-Flash-Vision-Exp. - Models "fill in from memory": Llama and Mistral invented Polish words („zdębła”, „jazın”, „gęsta”); on the whiteboard with glare all the top models reproduced the formula „sin²α+cos²α=1”, probably from knowledge rather than from reading. - Qwen3-VL-235B skipped the small line with the date on the cursive page. Seed-2.0-mini added a heading „### Prawa matematyczne” ("Mathematical laws"). - Generation speed: Qwen3-VL-235B/30B about 12–15 tokens/s (30–45 s per page); gemma-4-31B-it-turbo 48–112 s; Qwen3.6-35B-A3B 48–62 s to first token; gemma-4-26B-A4B 98–180 s; Qwen3.5-9B with reasoning enabled hit the token limit (150 s, CER 0.37). - The number of image tokens depends on the model: constant for Gemini (about 1305) and Seed (about 1515), saturating for Qwen3.5-35B (about 1170) and DeepSeek-V4.1-Flash (about 1189), constant and low for gemma-3/4 (about 256 – they lose small digits and subscripts), growing with resolution for Qwen3-VL, MiMo, Ling (Qwen3-VL-235B: 13884 tokens for 3276×4284 and 16568 for 7800×10150). - Formats and limits (platform behavior): MIME `image/jpg` and `image/heic` – HTTP 422; HEIC bytes labeled as jpeg – HTTP 400 on models hosted by DeepInfra (Gemini and Seed accept them); WebP – 200; a 21.8 MB PNG (29 MB request) accepted by Gemini, Qwen, gemma, rejected by Seed (10 MiB limit); a 79 Mpx image accepted by Gemini, Qwen3.5, gemma and Qwen3-VL-235B, rejected by Seed (36 Mpx limit). A rotated photo (EXIF 6) was read correctly by gemini, DeepSeek-V4.1, Seed, Qwen3.5, MiMo; Qwen3-VL-235B had diacW 0.81 on it. - JSON output can be dangerous for formulas: gemini-3.1-flash-lite wrote `\neq` without doubling the backslash, so after parsing it came out as a newline character + "eq" (3 of 3 attempts on `notes_photo`, also with a strict JSON schema); gemma-3-27b broke `\frac`, `\times`, `\rightarrow`. Markdown with a YAML header parsed 6/6 without damage. The instruction "return JSON" alone, without enforcing the format, gave valid JSON in 12 of 22 attempts; with enforcement (json_object) 21 of 22 (Ling once looped until the token limit); Seed-2.0-mini returned HTTP 500 with a strict JSON schema. - Image scaling (cursive page): capping the longer side at 2048 px did not lower the score for any model, and it prevented the growth in token count for Qwen3-VL, MiMo, Ling (3–10×). ## 2. "Hard" test from 2026-09-29/30: 14 photos, handwriting and small numbers - Platform: DeepInfra. - Data: 13 pages, each in a `normalne` (normal) and a `trudne` (hard) version (letters drawn separately with random slant, wavy paper, low resolution, strong blur and noise, JPEG 40–50). Pages: squared notebook (EXIF 6), lined notebook in cursive in poor light, chemistry with a table, whiteboard shot from the side with glare and a force diagram, chalkboard, printed worksheet with answers in pen, slide with integrals, PL/EN vocabulary (EXIF 8, motion blur), dense long page, cell diagram (EXIF 3), physics problems, test schedule, whiteboard with stoichiometry. The `trudne` set: 10 pages (105 facts; 210 over 2 runs) and separately 4 pages with small numbers (dense page, physics problems, schedule, stoichiometry). Rows from 2026-09-29: one run; the "separate measurement" row: 2 runs. - Facts: 135 expressions in the photos (total for the set). | model | CER (10 pages) | facts (10 pages) | CER / facts (4 pages with numbers) | time per photo, s | measurement date | |---|---|---|---|---|---| | deepseek-ai/DeepSeek-V4.1-Flash | 4.1% | 95.2% | 5.7% / 84% | 5–11 | 2026-09-29 | | Qwen/Qwen3.5-35B-A3B | 3.1% | 95.2% | 2.5% / 94% | 4 | 2026-09-29 | | moonshotai/Kimi-K2.6 | 2.5% | 97.1% | 2.3% / 91% | 11–19 | 2026-09-29 | | moonshotai/Kimi-K2.6 (separate measurement, 2 runs) | 2.8% | 96.7% (203/210) | 1.6% / 92.0% | 11.4 | 2026-09-30 | | Qwen/Qwen3.5-397B-A17B | 2.8% | 94.3% | 1.1% / 86% | 7–9 | 2026-09-29 | | XiaomiMiMo/MiMo-V2.6-Pro | 2.7% | 95.2% | 3.5% / 86% | 7–8 | 2026-09-29 | | Qwen/Qwen3-VL-235B-A22B-Instruct | 3.4% | 95.2% | 2.4% / 91% | 16–26 | 2026-09-29 | | Qwen/Qwen3.8-27B | 3.4% | 92.4% | n/a | 7 | 2026-09-29 | | MiniMaxAI/MiniMax-M3 | 6.4% | 92.4% | n/a | 10 | 2026-09-29 | On the `normalne` pages DeepSeek-V4.1-Flash read 8 of 10 pages without error (CER 2.2%, 104/105 facts). Differences between models show up mainly on the pages with small numbers in the `trudne` version. The most dangerous errors: on small handwriting DeepSeek-V4.1-Flash added lines that do not exist („Kartkówka: wtorek 14:05” – "Quiz: Tuesday 14:05" – instead of „Konsultacje: wtorek 14:30” – "Office hours: Tuesday 14:30") and recalculated an example by itself („r = 9”, „a₇ = 762”). **All** single models "corrected" the year 2026 to 2025 on the test-schedule page („05.11.2025”, „2025/2027”). ### Photo of a whole classroom with the board in the distance (2026-10-19, 1 photo) A board in the distance (about 8 px per character) read independently by three models: Kimi-K2.6 read formulas with an absolute value |x|; Qwen3.5-397B and Qwen3.5-35B "read" formulas for complex numbers that were not on the board; Qwen3.5-35B twice looped until the 6144-token limit. Agreement between the models' readings (trigram overlap): 0.95–0.99 for three pages of the `trudne` set, 0.33–0.41 for this distant photo. Conclusion: for small writing seen from a distance, models reconstruct the content instead of reading it. This is a single photo. ### Three hard photos of Polish notes, 2026-10-04 Synthetic fixtures, 3 photos, one run. | model | CER | facts | time per whole page | |---|---|---|---| | deepseek-ai/DeepSeek-V4.1-Flash | 4.5% | 90% | about 5 s | | inclusionAI/Ling-3.0-flash-VL | 7.8% | 82.5% | 25–265 s | | Qwen/Qwen3.5-397B-A17B (low reasoning) | 1.5–1.9% | 88–92% | 1–4 min | | Qwen/Qwen3.8-27B | 5.3–5.8% | 85–90% | about 20 s | | moonshotai/Kimi-K3 (variant described in the source as "off") | 0.4% | 100% | n/a | ## 3. Textbook pages (print, scans and photos), 2026-10-05 - Platform: DeepInfra. - Data: 7 "textbook-like" pages (history, quadratic function with formulas, solubility table with a reaction, physics in two columns with a figure, a poem, two dense pages of ~2000 characters – biology and sequences), each in 3 versions: 150 dpi scan, phone photo (skewed, shadow, noise, JPEG 72), "hard" photo (~1150 px, shadow across half the page, JPEG 55). 21 readings per model, 56 facts (numbers, formulas, table cells, reaction). The source gives results without confidence intervals. | model | CER scan | CER photo | CER hard | facts | printed page number | table as table | s/page | |---|---|---|---|---|---|---|---| | deepseek-ai/DeepSeek-V4.1-Flash | 0.10% | 0.13% | 0.38% | 99% | 100% | 3/3 | 6.8 | | inclusionAI/Ling-3.0-flash-VL | 0.08% | 0.11% | 0.26% | 100% | 100% | 3/3 | 8.0 | | Qwen/Qwen3.5-35B-A3B | 1.39% | 1.27% | 0.73% | 98% | 100% | 3/3 | 5.4 | | Qwen/Qwen3.8-27B | 0.22% | 0.20% | 0.36% | 96% | 100% | 3/3 | 12.1 | | Qwen/Qwen3.5-397B-A17B | 0.03% | 0.06% | 0.00% | 99% | 100% | 3/3 | 12.6 | | moonshotai/Kimi-K2.6 | 0.04% | 0.04% | 0.06% | 100% | 100% | 3/3 | 30.2 | On print all models are close to each other (CER below 1.5%, page number 100%). On hard **handwriting** (the notebook photos from 04.10, section 2) DeepSeek-V4.1-Flash was clearly better than Ling (CER 4.5% vs 7.8%). ### Cheaper models, 2026-10-06 (the same 21 readings) Note: DeepInfra silently **redirects retired models to a replacement**; the "responded" column is the `model` field from the response. | model (requested) | responded | CER scan | CER photo | CER hard | facts | page number | |---|---|---|---|---|---|---| | Qwen/Qwen3.5-0.8B | Qwen/Qwen3.5-9B | 0.31% | 0.64% | 0.51% | 98% | 90% | | Qwen/Qwen3.5-2B | Qwen/Qwen3.5-9B | 0.31% | 0.61% | 0.53% | 98% | 90% | | Qwen/Qwen3.5-4B | Qwen/Qwen3.5-9B | 0.31% | 0.64% | 0.53% | 98% | 90% | | Qwen/Qwen3.5-9B | Qwen/Qwen3.5-9B | 0.31% | 0.64% | 0.53% | 98% | 90% | | deepseek-ai/DeepSeek-OCR | google/gemma-4-31B-it | 0.55% | 0.48% | 1.12% | 96% | 100% | | allenai/olmOCR-2-7B-1025 | google/gemma-4-31B-it | 0.55% | 0.73% | 1.01% | 96% | 100% | | microsoft/Phi-4-multimodal-instruct | google/gemma-3-12b-it | 100% | 100% | 100% | 0% | 0% | | mistralai/Mistral-Small-3.1-24B-Instruct-2503 | Mistral-Small-3.2-24B | 0.95% | 0.98% | 0.91% | 99% | 100% | | mistralai/Mistral-Small-3.2-24B-Instruct-2506 | (the same) | 0.30% | 0.98% | 0.94% | 98% | 100% | Conclusions: "DeepSeek-OCR" and "olmOCR" on DeepInfra (as of 2026-10-06) are in fact gemma-4-31B-it (retired, with `replaced_by`); Qwen3.5-0.8B/2B/4B were answered by the 9B model (identical results), which got the page number wrong in 10% of readings; Mistral-Small had about 1% CER on photos. ### 305-page scan and loops on leader dots, 2026-10-08 Test pages like those of a WOS textbook (civics; two columns, boxes, a chart, photos, a dense page of ~4000 characters, a table of contents with leader dots, exercises with dotted lines), A4 scan at 300 dpi, real calls, small sample. | page | Ling-3.0-flash-VL | DeepSeek-V4.1-Flash | |---|---|---| | regular (701–1467 output tokens) | 8–62 s, sometimes 100–185 s (platform queue) | 7–27 s | | exercises with dotted lines | 8–100 s, passes | response judged as looped (15 identical lines of dots) | | table of contents with dots | loop „. . . .” until the 6144-token limit (88–149 s), 3/3 attempts | loop until 6144 tokens in one of the attempts (older prompt), 490 tokens in another | Ling: in this measurement it had no flex mode (the platform responded with `service_tier: default`). ## 4. Describing illustrations (charts, diagrams, maps), 2026-10-06 - Platform: DeepInfra. - Data: 9 drawings with deliberately made-up data (so that the model has to actually read them): a bar chart with values, a function graph with points without labeled coordinates, a circuit diagram, a photosynthesis diagram, a triangle with an altitude, a map of an island with a legend and scale, a plant cell with labels, a line chart without values (read from the grid), a circle with a tangent. Each in a clean version and as a crop from a photo (rotation, noise, blur, JPEG 70); plus a series logo (the correct answer is "decoration"). 75 facts (labels, numbers, axes, markings, relations); 18 descriptions per model. - No Google, Anthropic or Kimi models in this measurement. All models answered themselves (no redirects). | model | facts | clean | photo crop | full descriptions (of 18) | logo = "decoration" | s | |---|---|---|---|---|---|---| | Qwen/Qwen3.5-397B-A17B | 98.2% | 100% | 96% | 16 | Y | 23.8 | | Qwen/Qwen3.6-27B | 97.6% | 99% | 96% | 14 | Y | 146.6 | | ByteDance/Seed-2.0-pro | 96.3% | 95% | 98% | 13 | N | 7.0 | | Qwen/Qwen3-VL-235B-A22B-Instruct | 95.7% | 99% | 93% | 14 | Y | 15.2 | | thinkingmachines/Inkling | 95.1% | 96% | 94% | 11 | N | 5.2 | | XiaomiMiMo/MiMo-V2.6-Pro | 95.1% | 98% | 93% | 13 | Y | 8.5 | | meta-models/Muse-Glimmer-30B | 94.5% | 96% | 93% | 11 | Y | 33.3 | | MiniMaxAI/MiniMax-M3 | 93.9% | 94% | 94% | 13 | Y | 12.6 | | inclusionAI/Ling-3.0-flash-VL | 93.3% | 95% | 91% | 12 | N | 10.1 | | deepseek-ai/DeepSeek-V4-Flash-Vision-Exp | 92.7% | 95% | 90% | 12 | N | 4.9 | The only gaps of the best model: values from the line chart on the **rotated** photo (rotation shifts the points relative to the grid by about 2 °C – all models get this wrong). Qwen3.5-397B measured on a longer set of 16 descriptions: facts 143/144 (99.3%) in both runs. --- # OCR: exam-sheet photos with handwritten answers > Gemini 3.1 Pro read circled answers 80/80 and did not invent answers on blank pages (12/12), but misread a handwritten word toward the printed word on the same page depending on the prompt; Qwen3-VL-235B invented answers on a blank sheet during full-page reading yet answered a single yes/no "is there handwriting" question correctly. Samples are very small. Area: Vision & OCR. Page: https://benchmarks.site/ocr-exam-sheets/. Methodology and caveats: https://benchmarks.site/methodology/#exam-sheets # OCR: reading photos of exam sheets (vision models) All rows are **individual models** on a specific task with a photo: reading handwriting, circled answers, recognizing a blank page, transcribing print. The photos are pages of Polish school exam sheets. Samples are **very small** (one to a few photos, a few to a few dozen repetitions). There is no single "reading accuracy: X%" figure per model here, because nobody transcribed by hand and checked answers across many pages. Costs are the amounts reported by the provider itself in the `estimated_cost` field of the response (converted to grosze; 1 gr = 0.01 PLN), not calculations from the token price list. Metric definitions and caveats: [overview](/methodology/#exam-sheets). ## 1. Reading a single handwritten word (Gemini 3.1 Pro) - Model: `google/gemini-3.1-pro` via DeepInfra (forwarded to Google), reasoning at the `medium` level (according to the configuration from that period; reconstruction). - Data: **one photo** of a Polish-language exam-sheet page (2026), 1500×2000 px, with a short handwritten open answer. The word under study: the student wrote „opracował” ("worked out / developed"), while the printed text on the same page has „opanował” ("mastered"). The reference was checked by inspecting an enlarged crop of the photo (the first version of the reference was wrong and gave „opanował”). - "Prompt variant" means differences in the instruction and in the task description passed along with the photo; the prompt texts are not given here. | conditions | date | attempts | „opracował” (correct) | „opanował” (word from print) | |---|---|---|---|---| | prompt variant 1 | up to 2026-09-23 | 18 | **18/18** | 0 | | prompt variant 1, whole-page reading | 2026-10-13 | 20 readings (10 runs of 2) | 19/20 | 1/20 | | prompt variant 2 (different description of the tasks on the page) | 2026-10-13 | 10 readings (5 runs of 2) | 5/10 | 5/10 | | prompt variant 3 (with a request to correct spelling only) | 2026-10-13 | 3 | 0/3 | 3/3 | Observations: - The error always went toward the word printed on the same sheet. The model "fits" the handwriting to the text it sees in print. - In variant 3 the model, in 1 of 3 attempts, also changed a word form („tą” → „tę”, a grammatical form change), even though the prompt asked to correct spelling only. In approx. 80 other readings of this page such a change did not appear. - Aggregated (2026-10-13, 40 readings of this page in variant 1): the whole open answer (one sentence) matched the reference word for word in **33/40**, and with word agreement ≥ 90% in **40/40**. The figures were provided by an agent analyzing the saved readings; they may partly overlap with row 2 of the table. ## 2. Circled closed-ended answers (Gemini 3.1 Pro) - Model as in section 1, the same photo: two closed-ended tasks with a handwritten circle around the answer letter. - 2026-10-13: **80/80** correct letters (40 readings × 2 tasks). Each time the model described the mark the same way ("handwritten circle around the letter …") with confidence "high". ## 3. Blank exam-sheet page | model | data | date | attempts | result | |---|---|---|---|---| | google/gemini-3.1-pro (full page reading) | 4 photos of exam-sheet pages with no handwriting at all, 1500×2000 px | 2026-10-13 | 12 (3 per page; the agent reported 15 in total) | 12/12 with no invented answers | | Qwen/Qwen3-VL-235B-A22B-Instruct (full page reading; model reconstructed) | rendered blank mathematics exam sheet, 2 pages | 2026-09-04 | 2 runs | **10 invented answers** on 2 pages, described as confidently as real ones; the second run repeated the same error | | Qwen/Qwen3-VL-235B-A22B-Instruct (closed question: is there handwriting on the page) | the same blank sheet and the same pages with marks drawn in | 2026-09-04 | several (exact number unknown) | correct: "no handwriting" on blank pages, "handwriting present" on pages with marks | | Qwen/Qwen3-VL-235B-A22B-Instruct (the same closed question) | photo from section 1 (page with handwriting) | 2026-10-13 | 40 | 40/40 "handwriting present" | Observations: - The same model that invented marks on a blank sheet during full-page reading answered correctly when asked about a single thing (whether there is handwriting). - Models often claimed to see a "blue circle" on a sheet printed in black and white (qualitative observation from 2026-09-04, no attempt count). ## 4. Identifying the exam sheet from photos - Data: real photos of Polish-language 2026 exam-sheet pages; the exam-sheet code is usually not visible in the photo (it is in the footer). - Approx. 2026-09-22. | task | model | attempts | result | |---|---|---|---| | give the exam-sheet code visible in the photos | Qwen/Qwen3-VL-235B-A22B-Instruct, google/gemini-3.1-pro (combined) | 6 | **0/6 correct**, including 2 times an English-language exam sheet was indicated. Qwen repeated the example values given in the prompt, Gemini indicated an exam sheet from a different year | | transcribe a few sentences of print from each of 4 photos | both models | 7 photo sets | print transcribed usefully in all 7 sets (7/7; no CER measurement) | Observations: - A model asked for a code it cannot see gives something plausible-looking instead of "I don't know". - A code that was read can differ by one digit ("2505" read as "2605"); qualitative observation. - Transcribing whole dense pages of Polish-language print took over 3 minutes and ended in a timeout (the model is not stated in the source, most likely Qwen3-VL-235B). ## 5. Cost and time of a single call Costs from the provider's (DeepInfra) `estimated_cost` field, converted to grosze (gr; 1 gr = 0.01 PLN). Times from sending to the full response, no streaming, from Poland. | model | task | date | sample | cost per call | time per call | |---|---|---|---|---|---| | google/gemini-3.1-pro (reasoning `medium`) | full reading of a page with handwriting | 2026-09-17 | 5 photos of one exam sheet | approx. **12.3 gr** | n/a | | google/gemini-3.1-pro | full reading of a blank page | 2026-09-17 | same as above | approx. **3.2 gr** | n/a | | google/gemini-3.1-pro | average per page (mix of blank and written pages) | 2026-09-17 | same as above | approx. **6.9 gr** | n/a | | google/gemini-3.1-pro | full reading of a blank page (1500×2000 px) | 2026-10-13 | 12 | n/a | **6.3–19.7 s** (typically 15–20 s) | | google/gemini-3.1-pro | full reading of a page with handwriting (1500×2000 px) | 2026-10-13 | 13 | n/a | approx. 25–30 s (**indirect estimate**, a single call was not measured separately) | | Qwen/Qwen3-VL-235B-A22B-Instruct | closed question about the page (one-word answer) | 2026-10 | n/a | approx. **0.1 gr** | a few seconds (estimate) | | Qwen/Qwen3-VL-235B-A22B-Instruct | transcribing fragments of print from 4 photos | 2026-09-22 | 7 sets | approx. **0.50 gr** | n/a | | google/gemini-3.1-pro | transcribing fragments of print from 4 photos | 2026-09-22 | 7 sets | approx. **4.1 gr** | n/a | Note: for Gemini, calculating the cost from the token price list and the `completion_tokens` field underestimated the cost of a page reading **approx. 13-fold**, because reasoning tokens are billed but not included in `completion_tokens` (see [Platforms](/platforms-errors-billing-prices/)). ## 6. Qwen3-VL-235B-A22B "Thinking" on DeepInfra (2026-09-03) A request to the `Qwen3-VL-235B-A22B-Thinking` variant was served by the Instruct model: **0 reasoning tokens** in the response, at a rate approx. **2×** higher than for Instruct. Details in [Platforms](/platforms-errors-billing-prices/). An earlier estimate from the price list (before the measurement) assumed an approx. 6× higher cost for the Thinking variant. ## 7. Google Document AI (Enterprise Document OCR) No accuracy measurements in the sources. One early comparison with Gemini 3.1 Pro on one page was scored against the wrong reference (see section 1), so it settles nothing. Qualitative observation: the OCR transcribes everything in a given area, including the printed text of the exam sheet, so its reading of the area around an answer can be several times longer than the student's answer itself. Billing is per page (according to Google's documentation, not measured). --- # Language models: classification, summaries, chat with tools > DeepSeek-V4.1-Flash was the most consistent choice across lesson classification, faithful lesson summaries (99.6–99.7% of knowledge items kept, no truncated notes) and streaming tool chat. Abbreviations like EDB and WOS confuse cheap classifiers unless explained in the prompt, and reasoning toggles differ between models. Area: Language models. Page: https://benchmarks.site/llm-classification-summaries-chat/. Methodology and caveats: https://benchmarks.site/methodology/#speech-vision-text Platform: DeepInfra (unless noted otherwise). Results of single models on specific tasks. Latencies depended on the platform load at the time (queues and timeouts occurred). Prices omitted. Description of metrics and caveats: [project overview](/methodology/#speech-vision-text). ## 1. Lesson subject classification, 2026-09-22 - Task: from about 1100–1300 words of a lesson transcript (recognized speech + photo reading), pick the subject from a list of 15 labels (Matematyka, Fizyka, Chemia, Biologia, Geografia, Historia, Język polski, Język angielski, Język niemiecki, Informatyka, WOS, EDB, Religia, Wychowanie fizyczne, Inne – i.e. Mathematics, Physics, Chemistry, Biology, Geography, History, Polish, English, German, Computer science, civics, education for safety, Religion, Physical education, Other), the topic and a confidence. Strict JSON schema enforced, temperature 0. - Data: 6 cases – biology, Polish literature, physics (lots of math), mathematics (lots of physics examples), EDB/first aid (ambiguous with biology), an art lesson that should go to „Inne” (Other). About 1.1–2.5k input tokens per case. **Small sample: 6 cases, one call per case.** - "Without explanations" = cases 1–5 without an extra explanation of the abbreviations (EDB, WOS) in the prompt; "with explanations" = 6 cases with a short explanation of the abbreviations. | model | latency | valid JSON | correct without explanations (of 5) | correct with explanations (of 6) | |---|---|---|---|---| | openai/gpt-oss-20b | 1.0–1.9 s (one 3.6 s) | 100% (also without an enforced schema) | 4/5 (EDB → Biologia) | 6/6 | | openai/gpt-oss-120b | 1.9–3.4 s | 100% | 4/5 (EDB → WOS) | 6/6 | | deepseek-ai/DeepSeek-V4-Flash | 1.3–2.8 s (one 8.3 s) | 100% | 5/5 | 6/6 | | deepseek-ai/DeepSeek-V4.1-Flash | 1.8–5 s, one 180 s timeout | 100% | n/a | 6/6 | | Qwen/Qwen3-Next-80B-A3B | 0.9–2.5 s | 100% | 4/5 | 5/6 (art → „Język polski”, confidence 0.3; in json_object mode it returned a label outside the list: „Sztuka” – "Art") | | google/gemma-3-27b-it | 1.9–4 s | 100% with schema; without schema wraps the JSON in `` ```json `` | 4/5 | 5/6 (art → Język polski) | | meta-llama/Llama-4-Scout-17B-16E-Instruct | 1.2–2.4 s | 100% with schema; without schema adds a comment | 4/5 | 6/6 (typo „Resusytacja”) | | Qwen/Qwen3-32B | 17–22 s with thinking; 2.1–6.8 s with it disabled | 100% | 4/5 (EDB → Wychowanie fizyczne) | EDB correct | | zai-org/GLM-5.3-Flash | 6–12 s; later 75 s and a 180 s timeout | 100% | 5/5 | EDB correct | | google/gemma-4-26B-A4B-it | 17–32 s (1.4–4.5 s in a later attempt) | 100% | 4/5 | n/a | Observations: the abbreviations "EDB" and "WOS" confuse cheap models (the first-aid lesson ended up in Biology, WOS or PE); a one-sentence explanation of the abbreviations improved the result for all 8 tested. A strict JSON schema forces a label from the list; without it Qwen3-Next returned „Sztuka”. Gemma-3 and Qwen3-Next put the art lesson in „Język polski” even with the schema. Qwen3-32B requires disabling thinking mode, otherwise it takes 17–22 s to answer. ## 2. Lesson summary (description), 2026-09-22 - Task: from the speech transcript and photo reading, write a faithful description of the lesson in Polish. Data: 2 lessons – photosynthesis (2 recordings + a photo reading, about 2.3k input tokens) and „Dziady” part III (a Polish Romantic drama). Qualitative assessment by the author of the measurement, **2 lessons, 1–3 runs**. | model | first token | total | output tokens | quality assessment | |---|---|---|---|---| | deepseek-ai/DeepSeek-V4.1-Flash | 1.35–1.5 s | 16–37 s | 570–650 | best: faithful, complete, no invented facts, clean Polish | | deepseek-ai/DeepSeek-V4-Flash | 1.7 s | 15–21 s | about 500 | faithful, more concise | | Qwen/Qwen3.5-397B-A17B (thinking disabled) | 0.8–1.1 s | 17 s | about 600 | good; used the term „faza ciemna” ("dark phase") although the teacher advised against it; added one invented line about the next lesson; LaTeX | | Qwen/Qwen3-235B-A22B-2507 | 1.0–1.2 s | 30–40 s | n/a | invented facts („imię z 44 liter” – "a name of 44 letters", „np. Geniusz stróż” – "e.g. Guardian Genius") | | moonshotai/Kimi-K2.6 (default thinking) | none | 79 s | 3000 reasoning tokens, 0 content | empty result (token limit used up on reasoning) | | moonshotai/Kimi-K2.6 (thinking disabled) | 1.2 s | 23 s | n/a | good; one grammatical error | | deepseek-ai/DeepSeek-V3.2 | 5.3 s | 103 s (about 7 tokens/s) | n/a | faithful | | google/gemma-4-31B-it | 3.9 s | 107 s | n/a | faithful; LaTeX | ## 3. "Lossless" summaries: how much lesson knowledge survived, 2026-09-29 - Task: from a 45-minute noisy lesson transcript (fillers, repetitions, dictation with „przecinek” ("comma") spoken aloud, off-topic chatter, broken-off sentences, a whiteboard photo reading with `[?]`), write a full note that replaces the recording (metaphors, quotes, student questions, homework). The same prompt template for all models (version v2 or v3 depending on the row), one note per lesson. No Google models. - Data: 4 synthetic high-school lessons: mathematics (53 knowledge-list items), history (60), biology (57; three overlapping student recordings), Polish (60; a dense lesson, 28 min of speech). Input of 17–28k characters. Each list item is one checkable fact. - Scoring: a judge model (DeepSeek-V4-Pro-0813, temperature 0, every verdict with a verbatim quote, checked by code); "present" only when all details are there. The judge is rather too strict (e.g. "missing" for a student question when only the answer is there; "partial" for a surname already garbled in the input transcript). | model | present (share of items) | output tokens | time, s | |---|---|---|---| | deepseek-ai/DeepSeek-V4.1-Flash | 99.6–99.7% | 4070 | 56–99 | | deepseek-ai/DeepSeek-V4-Pro-0813 | 98.7% (v2 and v3) | 3950 | 45–53 | | moonshotai/Kimi-K2.6 | 98.3% | 4500 | 91 | | MiniMax-M3 | 95.7% | 3850 | 180 | | deepseek-ai/DeepSeek-V4-Flash | 93.9% (v2), 90.0% (v3) | 3400 | 86–98 | | Qwen/Qwen3.5-397B-A17B | 92.6% | 4400 | 240 | - DeepSeek-V4.1-Flash, all lessons: 98.7%, 99.6% and 99.7% in three successive versions of the prompt template (2 runs per lesson). **0 truncated notes** (`finish_reason: length`) out of about 100 notes and **0 significant fabrications** in 20 checked notes (2 judge flags were false alarms). - DeepSeek-V4-Flash loses whole examples (mathematics 77–85%: without the problems with specific sequences and without the derivation of the formula). - Items models usually miss: an example shown only on the board, student questions recorded as just the answer. - Time with 3 parallel calls 45–65 s, with 8 parallel calls 80–130 s (most calls first got 429 "Model busy"). - The model fixes mishearings („bazyliszków” → „bazylianów”, "basilisks" → "Basilians"), but a recognition error that changes the meaning (a lost „nie” – "not") is copied faithfully. - Limitations: one note per lesson and model, judge instability, one of the Polish-lesson evaluations was interrupted (no funds left on the account, HTTP 402). ### DeepSeek-V4-Pro vs V4.1-Flash on lesson description, 2026-11-05 3 lessons (mathematics, biology, history) from the same fixtures, **one run each** (one or two list items are more than a percentage point). Reasoning disabled (0 reasoning tokens for both). | model | present | present + partial/2 | output tokens | time, s | |---|---|---|---|---| | deepseek-ai/DeepSeek-V4-Pro | 97.6% (biology 94.7; history 100; mathematics 98.1) | 97.6% | 3204 | 50–70 | | deepseek-ai/DeepSeek-V4.1-Flash | 98.8% (100; 98.3; 100) | 99.4% | 3698 | 22–26 | V4-Pro did not retain more content than V4.1-Flash in this trial. V4-Pro generated about 60 tokens/s. ## 4. Chat with tools, 2026-09-22 - Task: two streaming turns with two tools (search the student's notes, and the lesson timetable). Scenario 1: a question that needs both tools at once („jaki mam jutro plan lekcji i co mam w notatkach o rzucie ukośnym z fizyki” – "what is my timetable tomorrow and what do I have in my notes on projectile motion from physics"). Scenario 2: a search that returns no results. All models answered in Polish; none invented note content in the no-results scenario. **Small sample: a few calls per model** (range in the table). - "Arguments" = whether the tool-call arguments are streamed in pieces or arrive whole. | model | parallel calls | arguments | turn 1: first token / first tool call | turn 2: first text / total | reasoning | |---|---|---|---|---|---| | DeepSeek-V4.1-Flash | Y | in pieces | 1.0–2.3 s / 1.3–2.5 s | 1.0–1.5 s / 3.1–4.6 s | none by default | | DeepSeek-V4-Flash | Y | in pieces | n/a / 1.1–1.8 s | 1.7–2.5 s / 5.9–10 s | none | | Qwen3.5-397B (thinking enabled) | Y | in pieces | 0.7 s / 4.2 s | 3.0 s / 9 s | Y (separate field) | | Qwen3.5-397B (thinking disabled) | Y | in pieces | n/a / 1.0–1.1 s | 0.8 s / 6.3–7.2 s | none | | Kimi-K2.6 (default) | Y | in pieces | 2.9 s / 11 s (once 115 s / 172 s) | 28.7 s / 35.9 s | Y | | Kimi-K2.6 (thinking disabled) | Y | in pieces | n/a / 1.1–2.6 s | 1.0–1.2 s / 5.5–8.4 s | none | | Kimi-K3 | Y | whole in one chunk | 18 s / 27 s | 1.5 s / 6.8 s | Y | | GLM-5.3 | Y | in pieces | 1.0–1.1 s / 1.4–1.5 s | 1.1–1.25 s / 2.7–3.2 s | Y | | DeepSeek-V3.2 | Y | whole in one chunk | n/a / 9.7–19.6 s | 1.2–1.7 s / 11–28 s | none | | gpt-oss-120b | N (one call per round; 3 rounds) | whole in one chunk | 0.7–0.8 s / 1.5–3.8 s | 1.1 s / 9.5 s | Y | | DeepSeek-V4-Pro | Y | in pieces | n/a / 1.2 s | 3.2 s / 7.7 s | none | | Qwen3.8-Flash | Y | in pieces | 1.6 s / 2.7 s | 4.0 s / 7.4 s | Y, cannot be disabled | | GLM-5.3-Flash | Y | in pieces | n/a / 147.6 s | 182 s / 206 s | n/a | Observations: - The "search with no results" case: DeepSeek-V4.1-Flash ran two alternative queries, invented nothing and suggested next steps (the best behavior); GLM-5.3 looped through 11 calls over 4 rounds and never answered; Qwen3.8-Flash made 8 calls. - Kimi-K3: the first attempt ended with HTTP 429 (`engine_overloaded`) after 4 retries; a later one worked. GLM-5.3-Flash: 75–206 s to first token and 180 s timeouts. DeepSeek-V4.1-Flash: one 180 s timeout in about 25 calls. - GLM-5.3 with thinking disabled "leaks" English reasoning into the answer; it also made a grammatical error („na sprawdzianu”, an incorrect case form for "at the test"). DeepSeek-V4-Pro was verbose (a markdown table, ignored the length limit). - Kimi-K2.6 with default thinking used up 3000 tokens on reasoning and returned empty content. - Stream format: subsequent tool-call chunks have `name`, `id`, `type` set to null; GLM-5.3 can return two calls in a single chunk; Kimi pads arguments with spaces and assigns IDs like `functions.nazwa:0` (`nazwa` = name); Qwen3.5 and DeepSeek-V3.2 send empty text ("\n\n") before the calls. - Reasoning toggles differ: Kimi-K2.6 ignores two alternative disabling parameters; Qwen3-32B requires a different parameter; Qwen3.8-Flash ignores all the ones tried; DeepSeek-V4.1-Flash does not think by default. ### Whiteboard photo in chat (1 photo, 1024×1024) Task: transcribe the problems from the board and solve x²−5x+6=0 and 2x²+3x−2=0. 1 attempt per model. | model | result | first token / total | input tokens | |---|---|---|---| | DeepSeek-V4.1-Flash | correct | 1.85 s / 5.0 s | 720 | | Qwen3.5-397B (thinking disabled) | correct | 2.5 s / 6.4 s | 1039 | | Kimi-K2.6 (default thinking) | correct | 18.3 s / 20.3 s | 1459 | | google/gemma-4-31B-it | correct | 3.2 s / 14.6 s | 333 | | DeepSeek-V4-Flash, Qwen3.8-Flash | HTTP 405 – the model does not accept images | – | – | Combining an image with tools worked on DeepSeek-V4.1-Flash (two parallel search calls with a correct query and subject). ## 5. "Search the web or not" decision in chat, 2026-11-06 35 questions (21 should trigger a search, 14 should not), tools with mock results, one run. Correct decision: DeepSeek-V4.1-Flash 30/35, DeepSeek-V4-Pro-0813 29/35; 0 false searches for both. Typical mistake: the model reached for the materials, got empty results and answered from memory. The result depends on how the instruction is worded, so it is only indicative. ## 6. External models in chat (Claude, GPT), 2026-11-06: platform behavior Observations from live calls: - Claude (Sonnet and Opus) called through DeepInfra did not report cache hits with a long, repeated request prefix (0 hits). - GPT-6 (Responses API): the temperature parameter causes a 400 error; changing the reasoning level between requests invalidated the cache (0 hits despite an identical prefix). - The same text (tool definitions) is about 1.9× more tokens for Claude than for GPT-6. --- # Language models: short answers, grading text, time and cost > On DeepInfra, the undated name deepseek-ai/DeepSeek-V4-Flash hit an overloaded pool (7/12 requests rejected with HTTP 429), while the dated DeepSeek-V4-Flash-0731 answered every time and faster; automatic prompt caching cut the cost of a long step approx. 3.6×. No grading-accuracy test was run. Area: Language models. Page: https://benchmarks.site/llm-short-answers-grading-cost/. Methodology and caveats: https://benchmarks.site/methodology/#exam-sheets # Language models: answering questions, grading text, times and costs Individual models on text tasks: short answers to questions about a web service (with a few documentation fragments in context), tool calls in a conversation, grading students' answers. **There are no grading-accuracy measurements here** (agreement of the model's grade with an examiner's) – no such test was carried out. Costs from the provider's `estimated_cost` field. Caveats: [overview](/methodology/#exam-sheets). ## 1. DeepSeek-V4-Flash: undated name vs dated name (2026-10-13) - Platform: DeepInfra. Task A: a short "Say OK." (8 input tokens, 3 output tokens). Task B: a step of a conversation with tools, approx. 5.7 thousand input tokens, approx. 30 output tokens (a tool call). | model | task | attempts | result | time per call | cost per call | |---|---|---|---|---|---| | deepseek-ai/DeepSeek-V4-Flash (undated) | A | 12 | **7/12 rejected** (HTTP 429 `engine_overloaded`, "Model busy, retry later") | 8–13 s (those that went through) | 1.26·10⁻⁶ $ | | deepseek-ai/DeepSeek-V4-Flash-0731 | A | 6 | 6/6 | **1.2–2.3 s** | 1.02·10⁻⁶ $ | | deepseek-ai/DeepSeek-V4-Pro-0813 | A | 4 | 4/4 | **0.5–1.3 s** | n/a | | deepseek-ai/DeepSeek-V4-Flash (undated) | B | approx. 8 successful | some requests rejected as above | 7.3–15.8 s | 5.24·10⁻⁴ $ without cache, 1.19·10⁻⁴ $ with a cache hit | | deepseek-ai/DeepSeek-V4-Flash-0731 | B | approx. 17 | all went through | **5.2–12.6 s** (typically 6–9 s) | 3.5·10⁻⁴ $ without cache, **9.8·10⁻⁵ $** with a cache hit (5632 of 5763 tokens from cache) | Observations: - The same model under the undated name sat on an overloaded pool: roughly every second request came back with 429, and the same request retried a second later usually went through. The dated version responded every time. - 0 reasoning tokens for both names (no reasoning parameter). - A repeated long prompt hit the cache automatically; the cost of the step dropped approx. 3.6×. ## 2. Short answers from documentation (gemma-4-E4B-it, 2026-09-04) - Model: `google/gemma-4-E4B-it` via DeepInfra. Task: answering a question about a web service in Polish, with at most 3 short document fragments in context. - Cost: approx. **0.01 gr per question** (1 gr = 0.01 PLN; measured; the number of questions in the sample is unknown). - The model does not accept photos in this use case (not tested). - On 2026-10-16 the model was not on DeepInfra's `/models` list, but the name was recognized (see [Platforms](/platforms-errors-billing-prices/)). ## 3. Grading students' answers and hints: costs See [Reasoning](/reasoning-effort/), section 2: DeepSeek-V4-Flash with reasoning 7.3× more expensive than without; DeepSeek-V4-Pro with reasoning approx. 40× (hint) compared with Flash without reasoning; grading a short answer worth 2 points with DeepSeek-V4-Pro: 1914 reasoning tokens, 3.38 gr. Operational run (2026-09-04): a full mathematics exam sheet graded by a model (DeepSeek-V4-Flash-0731) completed without errors; the score of 14/30 points applies to a specific test paper and **is not a measure of accuracy**. ## 4. Models without measurements - `anthropic/claude-opus-5-5` (DeepInfra): neither quality, cost nor time was measured (on 2026-10-16 the account returned 402). Price from the list in [Platforms](/platforms-errors-billing-prices/). - Kimi, GLM, text Qwen3.x models, gpt-oss: no measurements in these measurements. --- # Reasoning level (reasoning_effort): what models actually do > On `minimal`, only DeepSeek-V4-Pro actually produced reasoning tokens; gpt-5 accepts `minimal` but does not reason and gpt-5.6 rejects it. Reasoning is expensive (DeepSeek-V4-Flash 7.3× more with reasoning) and can consume the whole output limit, yielding empty responses (gpt-5.6 at a 2000-token limit). Area: Language models. Page: https://benchmarks.site/reasoning-effort/. Methodology and caveats: https://benchmarks.site/methodology/#exam-sheets # Reasoning level (`reasoning_effort`): what models actually do Measurements of whether a given `reasoning_effort` value produces real reasoning tokens, and what it costs. "Reasons" = the response reports a non-zero number of reasoning tokens (`reasoning_tokens` or a separate field with the reasoning content). Samples are small: a few calls per model, on tasks of grading students' answers and giving hints for exam tasks. Caveats: [overview](/methodology/#exam-sheets). ## 1. The `minimal` value on different models (2026-09-03) | model | platform | `minimal` accepted? | reasons on `minimal`? | notes | |---|---|---|---|---| | deepseek-ai/DeepSeek-V4-Pro (`-0813`) | DeepInfra | Y | **Y** | the only model in the test that actually reasoned on `minimal` | | deepseek-ai/DeepSeek-V4-Flash (`-0731`) | DeepInfra | Y | **N** | response longer than without the parameter, but 0 reasoning tokens; reasons only from `low` | | gpt-5 | OpenAI | Y | **N** (0 reasoning tokens) | lowest level with real reasoning: `low` | | gpt-5.6 (variants `terra`, `sol`) | OpenAI | **N** (value rejected) | – | lowest accepted level: `low` | | anthropic/claude-opus-5-5 | DeepInfra | per Anthropic documentation, N (lowest is `low`) | – | **not measured live** (on 2026-10-16 the account returned 402) | ## 2. Cost of reasoning | model | task | date | sample | result | |---|---|---|---|---| | deepseek-ai/DeepSeek-V4-Flash | hint for an exam task, with reasoning (`low`) vs without | 2026-09-03 | a few calls | with reasoning **7.3×** more expensive than without | | deepseek-ai/DeepSeek-V4-Pro | the same task, with reasoning, compared with DeepSeek-V4-Flash without reasoning | 2026-09-03 | a few calls | approx. **40×** more expensive | | deepseek-ai/DeepSeek-V4-Pro (model reconstructed from change history) | grading a short open answer worth 2 points, with reasoning | 2026-09-03 | 1 task | **1914 reasoning tokens, 3.38 gr** (1 gr = 0.01 PLN), i.e. roughly the same as grading an essay worth 20 points | Observation: the cost of reasoning hardly depends on the length of the task. A short answer worth 2 points used a similar number of reasoning tokens as a long essay. ## 3. Reasoning and the output token limit | model | conditions | date | result | |---|---|---|---| | gpt-5.6-terra / gpt-5.6-sol | lowest accepted reasoning level, output limit of 2000 tokens, task with a photo | 2026-09-03 | **empty response**: reasoning used up the whole limit before the model wrote a single character; a limit of around 6000–8000 was needed | | Qwen/Qwen3-VL-235B-A22B-Instruct | reasoning `low`, reading a page from a photo | 2026-09-13 | reasoning tokens count toward the same limit as the response, so the limit has to be raised (approx. +1500 tokens was adopted); this is an assumption based on the behavior of other models, an empty response from Qwen was not recorded | Qualitative observation (2026-09-13, no numbers): Qwen3-VL-235B with `low` reasoning confused smudged boxes and crossed-out answers less often than without reasoning. This was not measured on a set with a reference. ## 4. Billing of reasoning tokens With `google/gemini-3.1-pro` (DeepInfra → Google), reasoning tokens are billed but not included in `completion_tokens`. The cost calculated from the price list and `completion_tokens` came out approx. **13× too low** compared with `estimated_cost` (2026-09-17, reading a page from a photo, 5 photos). --- # Embeddings: retrieval accuracy in Polish > On a small Polish note set, the Qwen3-Embedding models (0.6B, 4B, 8B) ranked relevant notes above lookalike distractors, with Qwen3-Embedding-0.6B giving the best margin and 10/10 queries with an instruction prefix; bge-m3, multilingual-e5-large and embeddinggemma-300m let unrelated notes outrank a relevant one. Area: Embeddings. Page: https://benchmarks.site/embeddings-polish/. Methodology and caveats: https://benchmarks.site/methodology/#speech-vision-text Platform: DeepInfra. Measurement: 2026-09-22. Description of metrics and caveats: [project overview](/methodology/#speech-vision-text). ## Data - 12 Polish note documents, including deliberately misleading, off-topic ones: „równania reakcji” ("reaction equations"), „równania Maxwella” ("Maxwell's equations"), „układy równań liniowych” ("systems of linear equations"), „kwadratura koła” ("squaring the circle"). - Graded test for the query „równania kwadratowe” ("quadratic equations"): a point score (0–1 scale; the exact definition is not given in the source) and a **margin** = lowest score of a relevant note minus highest score of an irrelevant note (a negative value = an irrelevant note outranked a relevant one). - 10-query test: point score 0–1 (1.00 = full accuracy); for bge-m3 the source gives 0.95 and 9 of 10 queries with a relevant note in first place. - **Small sample**: 12 documents, 10 queries, one call per configuration. - "With instruction" = the query is prefixed with a short task instruction (recommended for the Qwen3-Embedding family; documents without a prefix). | model | dimensions | latency per batch | graded test „równania kwadratowe”: score / margin | 10-query test | |---|---|---|---|---| | BAAI/bge-m3 | 1024 | 1.5–1.6 s | 0.632 / −0.063 | 0.95 (9/10 in first place) | | Qwen/Qwen3-Embedding-0.6B | 1024 | 1.5–3 s (one cold 26 s) | 0.953 / +0.179 without instruction; 0.935 / +0.187 with instruction | 0.925 without instruction; 1.00 with instruction | | Qwen/Qwen3-Embedding-4B | 2560 | 1.5–5.8 s | 0.953 / +0.078; with instruction margin +0.127; at 1024 dimensions 0.888 / +0.104 | 1.00 | | Qwen/Qwen3-Embedding-8B | 4096 | 10–31 s | 0.953 / +0.048; with instruction margin +0.097 | 1.00 | | multilingual-e5-large | 1024 (512-token limit) | 1.6–29 s | 0.751 / −0.030 (similarities clustered in the 0.73–0.88 range) | 1.00 | | embeddinggemma-300m | 768 | 0.9–1.6 s | 0.817 / −0.036 | 1.00 | Observations: - bge-m3 ranked the note with Viète's formulas below three unrelated notes that only share the word „równania” ("equations") or „kwadrat” ("square"). - Qwen3-Embedding-0.6B had the best margin (+0.19) on the graded test and 10/10 with the instruction. - Qwen3-Embedding-4B honors the dimension-reduction parameter (down to 1024). - Vectors from different models are not interchangeable. --- # Platforms: availability, latency and limits > Always check the `model` field in responses (DeepInfra silently redirects retired models to replacements), expect load-dependent latency and queues on DeepInfra, and note Soniox's default limit of 10 parallel streams per organization; OpenAI live transcription provides no end-of-utterance marker or word timestamps. Area: Platforms. Page: https://benchmarks.site/platforms-availability-latency-limits/. Methodology and caveats: https://benchmarks.site/methodology/#speech-vision-text # Platforms: behavior, availability, limits Observations about the platforms (DeepInfra, Soniox, OpenAI), collected while measuring models (September–November 2026). This is the state as of the given dates; platforms change, and load affects latency. Metric definitions and caveats: [overview](/methodology/#speech-vision-text). ## DeepInfra | observation | date | notes | |---|---|---| | Queue/overload of Qwen3-ASR-1.7B: for approx. 25 min `429 Model busy` responses, then for approx. half an hour a queue of 114–186 s | 2026-09-29 | a single episode | | Whisper and other models: latency depends on load, e.g. whisper-large-v3 up to 23 s for a 48 s clip; Qwen3-ASR 4–22 s for 10 min of audio and 15–27 s for 46 min | 2026-09-22 | | | Qwen3-ASR, 20 s chunk: 50–77 s under platform load (Nemotron 1.4–3.5 s) | 2026-11-05 | normal mode and `priority` gave similar times | | `flex` tier (lowest priority): on a short query, time to first response for DeepSeek-V4-Pro went from 1.3 s to 13–20 s, for DeepSeek-V4.1-Flash from 0.8–1.0 s to 3.6–4.8 s; according to a later assessment, `flex` latency was 8–40 s higher than in normal mode even without a queue | 2026-09-30 | speech models (Qwen3-ASR, Whisper, Nemotron) accepted this field with no effect | | Chat models under load: Kimi-K3 HTTP 429 `engine_overloaded` with 4 retries over approx. 30 s; GLM-5.3-Flash 75–206 s to first token and 180 s timeouts; Kimi-K2.6 once 115 s to first token; DeepSeek-V4.1-Flash one 180 s timeout in approx. 25 calls | 2026-09-22 | | | Retired models are silently redirected to a replacement (`replaced_by`), so "DeepSeek-OCR" and "olmOCR" responded as `google/gemma-4-31B-it`, small Qwen3.5 models (0.8B–4B) as Qwen3.5-9B, and Phi-4-multimodal as gemma-3-12b | 2026-10-06 | results require checking the `model` field in the response | | The cost field in the native ASR interface has inconsistent units: cents for Whisper and Voxtral, dollars for Qwen and Nemotron | 2026-09-22 | | | Images: MIME `image/jpg` and `image/heic` – HTTP 422 with a misleading message; HEIC bytes labeled as jpeg – HTTP 400 on models hosted by DeepInfra, while Gemini and Seed accept them; the documented 20 MB limit is not enforced (a 21.8 MB PNG was accepted); ByteDance Seed-2.0-mini has its own limits of 10 MiB and 36 Mpx (reported as HTTP 500) | 2026-09-22 | | | Video: the `video_url` part passes only the image to the model (see [video-audio](/omni-video-audio/)); audio input has a limit of 600 s per part | 2026-09-23 | | | Prompt caching: a repeated query with the same image and prompt is sometimes billed cheaper (automatic cache hits, varying between models) | 2026-09-22 | | | Claude (Sonnet, Opus) via DeepInfra: no reported cache hits | 2026-11-06 | see [LLM text](/llm-classification-summaries-chat/), section 6 | | Gemini models are forwarded to Google (partner flag); `gemma-4-31B-it-Ultra` is also a partner model | 2026-09-23 | | ## Soniox (live and file speech recognition) - Limit of **10 parallel streams per organization** (default); with 12 streams, 2 received 429 (2026-11-06). - With a congested uplink (12 streams + parallel file uploads), the first result took up to 7.7 s and streams were dropped (6 at once at second 46, including 3 with `408 Request timeout` and 3 without an error; one at second 63 without an error). Without other traffic, 30 of 30 streams ran to completion. The cause was not proven (the client-side link is the most likely). - The final result (`is_final`) in continuous speech arrived approx. 5.7 s after the word; the partial results are the useful ones. - Latin is not supported (a language hint including Latin gives HTTP 400 for the whole request). - Polish recordings, two model releases: `stt-rt-v3` is an alias of `stt-rt-v5`; the newest `stt-rt-v6` gave approx. 1 point lower WER, but the first result approx. 300 ms later ([ASR](/asr-polish/), section 5). ## OpenAI (transcription models) - `gpt-live-transcribe` and `gpt-realtime-whisper` in live mode send text deltas without an end-of-utterance marker or word timestamps, and reject server-side voice activity detection (2026-11-06). - The `gpt-4o-*` models are being shut down by OpenAI on 2027-02-26. - GPT-6 (Responses API): the temperature parameter causes error 400; changing the reasoning level between queries invalidates the cache (2026-11-06). --- # Platforms: errors, reasoning billing and prices > On DeepInfra, check the `model` field and reasoning-token counts (a "Thinking" variant was served by Instruct), prefer dated model names (undated DeepSeek-V4-Flash was often overloaded), and do not trust price-list cost estimates for Gemini (approx. 13× too low); 402/404/429 codes and error text distinguish account limits, missing models and pool overload. Area: Platforms. Page: https://benchmarks.site/platforms-errors-billing-prices/. Methodology and caveats: https://benchmarks.site/methodology/#exam-sheets # Platforms: behavior, availability, prices Observations about platforms (DeepInfra, OpenAI, Google), collected during model measurements (September–October 2026). This is the state on the stated dates; platforms change, and load affects latency. Caveats: [overview](/methodology/#exam-sheets). ## DeepInfra | observation | date | notes | |---|---|---| | The `Qwen/Qwen3-VL-235B-A22B-Thinking` variant was served by the Instruct model: 0 reasoning tokens in the response, at a rate approx. 2× higher than for `Qwen3-VL-235B-A22B-Instruct` | 2026-09-03 | worth checking the `model` field and the number of reasoning tokens in the response | | With missing balance, every call (including partner models, e.g. Gemini, and a request for 1 token) ends with HTTP 402 `You need positive balance to do inference. Please add balance manually or setup top-up` | 2026-10-13 … 2026-10-16 | the state lasted at least 3 days; the balance ran out during a series of approx. 140 test calls (cause unconfirmed) | | A model name that does not exist gives HTTP 404 `model_not_found` even with zero balance, while an existing name gives 402; so the response code tells you whether the model exists | 2026-10-16 | 3 names checked | | `/v1/openai/models` returned 180 models; `google/gemma-4-E4B-it` was not on the list, but the name was recognized (402, not 404), and in September the model responded normally | 2026-10-16 | the list is not a complete inventory of callable models | | `deepseek-ai/DeepSeek-V4-Flash` (undated name): approx. every second request HTTP 429 `engine_overloaded` "Model busy, retry later", a retry after a second usually went through; `deepseek-ai/DeepSeek-V4-Flash-0731` (dated) responded every time in 1.2–2.3 s | 2026-10-13 | 12 + 6 calls; details in [LLM text](/llm-short-answers-grading-cost/), section 1 | | 429 with `engine_overloaded` / "Model busy" in the body means overload of the model pool, not an account limit; 429 with a body about the request rate limit, and 402, are account limits | 2026-10-13 | distinguishable only by the error body | | Gemini (`google/gemini-3.1-pro`) bills reasoning tokens that are not in `completion_tokens`; the cost from the token price list came out approx. 13× too low compared with `estimated_cost` | 2026-09-17 | reading pages from photos, 5 photos | | Automatic prompt caching: with a repeated prompt of approx. 5.7 thousand tokens, `DeepSeek-V4-Flash-0731` returned `cached_tokens` 5632 and an approx. 3.6× lower cost | 2026-10-13 | without any parameter in the request | | Gemini and Claude (`anthropic/*`) models are available through the same DeepInfra key (forwarded to Google / Anthropic) | 2026-09-13, 2026-10-16 | | | The `anthropic/claude-opus-5-5` model page does not document support for `temperature`, `reasoning_effort` or `response_format` in the OpenAI-compatible API (the example contains only `max_tokens`) | 2026-10-16 | live behavior not checked (402) | ### Prices from `/v1/openai/models` (2026-10-16) USD per million tokens. "cache" = cache read, if given. Tags from the same list. | model | input | output | cache | tags | |---|---|---|---|---| | anthropic/claude-fable-5 | 10.00 | 50.00 | – | chat, vlm, vision, reasoning_effort, reasoning | | anthropic/claude-opus-5-5 | 4.00 | 20.00 | 0.20 | chat, vlm, vision, prompt_cache, reasoning_effort, reasoning; context 1000000 | | anthropic/claude-sonnet-5-5 | 2.00 | 10.00 | – | chat, vlm, vision, reasoning_effort, reasoning | | moonshotai/Kimi-K3 | 2.85 | 14.25 | 0.285 | chat, vlm, vision, prompt_cache, reasoning | | google/gemini-3.1-pro | 2.00 | 12.00 | – | chat, vlm, vision, reasoning_effort, reasoning; context 1000000 | | deepseek-ai/DeepSeek-V4-Pro-0813 | 1.30 | 2.60 | 0.10 | chat, prompt_cache | | google/gemini-3.5-flash | 1.50 | 9.00 | – | context 1000000 (tags not read) | | google/gemini-3.7-flash | 0.75 | 3.75 | – | chat, vlm, vision, reasoning_effort, reasoning | | deepseek-ai/DeepSeek-V4-Flash-Vision-Exp | 0.44 | 1.32 | 0.014 | chat, vlm, vision, prompt_cache, reasoning | | Qwen/Qwen3-VL-235B-A22B-Instruct | 0.20 | 0.88 | 0.11 | chat, vlm, vision, prompt_cache, reasoning_effort | | google/gemma-4-31B-it | 0.20 | 0.40 | – | chat, vlm, vision, reasoning | | Qwen/Qwen3.6-35B-A3B | 0.10 | 0.95 | 0.10 | chat, vlm, vision, prompt_cache, reasoning_effort, reasoning | | google/gemma-4-26B-A4B-it | 0.07 | 0.34 | – | chat, vlm, vision, reasoning; context 262144 | | deepseek-ai/DeepSeek-V4-Flash-0731 | 0.06 | 0.18 | 0.015 | chat, prompt_cache, reasoning | Note: list prices do not cover reasoning tokens that are not visible in `completion_tokens` (see above), so the real cost can be higher. ## OpenAI - `gpt-5`: `reasoning_effort: "minimal"` is accepted but gives 0 reasoning tokens (2026-09-03). - `gpt-5.6` (`terra`, `sol`): the `minimal` value is rejected; the lowest accepted level is `low` (2026-09-03). - `gpt-5.6-terra` / `gpt-5.6-sol` with an output limit of 2000 tokens and the lowest reasoning level returned empty content; a limit of approx. 6000–8000 was needed (2026-09-03). - See [Reasoning](/reasoning-effort/), section 1. ## Google Cloud Document AI (Enterprise Document OCR) - No accuracy or timing measurements in the sources. Billing per processed page (documentation, not measured). - Qualitative observation: the OCR returns all the text in the area, including the printed text of the exam sheet, so the reading of the area around a student's answer can be several times longer than the answer itself (see [OCR](/ocr-exam-sheets/), section 7). --- # Malware detection: AMTSO, EICAR containers, Objective-See hashes, Mac malware families > All 11 official AMTSO / eicar.org test files and EICAR in all 57 container formats are detected with no alarm on the 57 clean twins; coverage of real Mac malware is much lower — 8/33 Objective-See hashes are in the database and 57/83 families (2019–2025) have a family-named rule. Area: Security (macOS malware scanning). Page: https://benchmarks.site/malware-detection-safe/. Methodology and caveats: https://benchmarks.site/methodology/#malware Run `20261111`, 2026-11-11 17:15, total time 151.7 s. The benchmark uses only harmless public test files (EICAR test string, AMTSO Security Features Check files) and public metadata (SHA-256 hashes published in Objective-See reports, malware family names), so it can be run and published safely. | Part | Passed | Total | Rate | |---|---|---|---| | Official AMTSO Security Features Check + eicar.org files | 11 | 11 | 100% | | EICAR in container formats (must be detected) | 57 | 57 | 100% | | Clean twins of the same containers without EICAR (must not alarm) | 57 | 57 | 100% | | SHA-256 hashes from Objective-See reports 2019–2024 (database lookup only) | 8 | 33 | 24.2% | | Mac malware families 2019–2025 with a rule named after the family | 57 | 83 | 68.7% | In the tables below, "Detecting engines" lists the detection names recorded for the file, as `engine:signature`. At most four names are stored per file, so the list can be incomplete. Engine prefixes: `ClamAV` (ClamAV signatures), `XProtect` (Apple XProtect rules), `YARA-ext` (external YARA rule sets, e.g. BinaryAlert, Trellix ARC), `Heur` (the scanner's own heuristics), `MalwareBazaar` (signatures derived from MalwareBazaar data), `Fuzzy` (TLSH similarity match). ## 1. Official test files: AMTSO Security Features Check + eicar.org 11/11 detected. All are rated `MALICIOUS`, including the PUA test file and the cloud-lookup test (CloudCar), which the scanner detects with local signatures. | Test | Expected | Verdict | OK | Detecting engines (signature names) | |---|---|---|---|---| | eicar.org eicar.com | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test`, `YARA-ext:BINARYALERT_Eicar_Substring_Test` | | eicar.org eicar.com.txt | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test`, `YARA-ext:BINARYALERT_Eicar_Substring_Test` | | eicar.org eicar_com.zip | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `MalwareBazaar:Generic.zip`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test` | | eicar.org eicar_com2.zip (zip in zip) | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `MalwareBazaar:Generic.zip`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test` | | AMTSO download of malware (eicar.com) | malicious | MALICIOUS | Y | `YARA-ext:BINARYALERT_Eicar_Av_Test`, `YARA-ext:BINARYALERT_Eicar_Substring_Test`, `YARA-ext:TRELLIX_ARC_Malw_Eicar` | | AMTSO compressed malware (eicar.zip) | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test`, `YARA-ext:BINARYALERT_Eicar_Substring_Test` | | AMTSO compressed malware (eicar.zipx) | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test`, `YARA-ext:BINARYALERT_Eicar_Substring_Test` | | AMTSO compressed malware (eicar.exe, self-extracting) | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `Heur:PE.Toolchain.WinRAR-SFX`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test` | | AMTSO compressed malware (eicar_zip.exe, self-extracting) | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `Fuzzy:TLSH.njrat`, `Heur:PE.Toolchain.WinRAR-SFX`, `XProtect:EICAR` | | AMTSO potentially unwanted application | pua | MALICIOUS | Y | `ClamAV:PUA.Win.Trojan.Agent-1350158`, `ClamAV:Win.Tool.EICAR-9917185-0`, `Heur:PE.Checksum.Mismatch` | | AMTSO cloud lookup (CloudCar) | detected | MALICIOUS | Y | `ClamAV:Win.Trojan.11584714-1`, `Heur:PE.Checksum.Mismatch` | ## 2. EICAR in 57 container formats The EICAR test string packed into 57 formats: plain files, archives (zip with different compression methods and nesting up to 8 levels, gzip, bzip2, xz, lzma, zstd, lz4, Unix compress, tar variants, cpio, ar, 7z, xar), password-protected archives, package formats (deb, jar, nupkg, crx, asar, macOS `.pkg`, app bundle), office and e-book formats (docx, xlsx, odt, epub), e-mail (attachment, zip attachment, mbox), PDF attachments, base64, disk images (DMG in five formats, ISO 9660) and combinations (zip inside DMG, DMG inside zip). 57/57 were rated `MALICIOUS`. Detection count by engine prefix across sections 1 and 2 (68 files, up to four names each): `YARA-ext` 118, `ClamAV` 67, `XProtect` 64, `Heur` 15, `MalwareBazaar` 2, `Fuzzy` 1. | Test | Expected | Verdict | OK | Detecting engines (signature names) | |---|---|---|---|---| | plain .com | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test`, `YARA-ext:BINARYALERT_Eicar_Substring_Test` | | plain + CRLF (AMTSO style) | malicious | MALICIOUS | Y | `YARA-ext:BINARYALERT_Eicar_Av_Test`, `YARA-ext:BINARYALERT_Eicar_Substring_Test`, `YARA-ext:TRELLIX_ARC_Malw_Eicar` | | plain .txt | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test`, `YARA-ext:BINARYALERT_Eicar_Substring_Test` | | plain, no extension | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test`, `YARA-ext:BINARYALERT_Eicar_Substring_Test` | | zip (deflate) | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test`, `YARA-ext:BINARYALERT_Eicar_Substring_Test` | | zip (stored) | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test`, `YARA-ext:BINARYALERT_Eicar_Substring_Test` | | zip (bzip2) | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test`, `YARA-ext:BINARYALERT_Eicar_Substring_Test` | | zip (lzma) | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test`, `YARA-ext:BINARYALERT_Eicar_Substring_Test` | | zip in folder inside zip | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test`, `YARA-ext:BINARYALERT_Eicar_Substring_Test` | | zip nested 2× | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test`, `YARA-ext:BINARYALERT_Eicar_Substring_Test` | | zip nested 3× | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test`, `YARA-ext:BINARYALERT_Eicar_Substring_Test` | | zip nested 5× | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test`, `YARA-ext:BINARYALERT_Eicar_Substring_Test` | | zip nested 8× | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test`, `YARA-ext:BINARYALERT_Eicar_Substring_Test` | | gzip | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test`, `YARA-ext:BINARYALERT_Eicar_Substring_Test` | | bzip2 | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test`, `YARA-ext:BINARYALERT_Eicar_Substring_Test` | | xz | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test`, `YARA-ext:BINARYALERT_Eicar_Substring_Test` | | lzma (alone) | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test`, `YARA-ext:BINARYALERT_Eicar_Substring_Test` | | tar | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test`, `YARA-ext:BINARYALERT_Eicar_Substring_Test` | | tar.gz | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test`, `YARA-ext:BINARYALERT_Eicar_Substring_Test` | | tar.bz2 | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test`, `YARA-ext:BINARYALERT_Eicar_Substring_Test` | | tar.xz | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test`, `YARA-ext:BINARYALERT_Eicar_Substring_Test` | | cpio (newc) | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test`, `YARA-ext:BINARYALERT_Eicar_Substring_Test` | | ar | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test`, `YARA-ext:BINARYALERT_Eicar_Substring_Test` | | deb (ar + tar.gz) | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test`, `YARA-ext:BINARYALERT_Eicar_Substring_Test` | | jar | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test`, `YARA-ext:BINARYALERT_Eicar_Substring_Test` | | docx (OOXML part) | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `Heur:Office.EmbeddedExecutable`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test` | | xlsx (OOXML part) | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `Heur:Office.EmbeddedExecutable`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test` | | odt (ODF) | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test`, `YARA-ext:BINARYALERT_Eicar_Substring_Test` | | epub | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test`, `YARA-ext:BINARYALERT_Eicar_Substring_Test` | | crx (Chrome extension) | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test`, `YARA-ext:BINARYALERT_Eicar_Substring_Test` | | nupkg | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test`, `YARA-ext:BINARYALERT_Eicar_Substring_Test` | | asar (Electron) | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test`, `YARA-ext:BINARYALERT_Eicar_Substring_Test` | | e-mail attachment (base64) | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `Heur:Email.ExecutableAttachment`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test` | | e-mail with zip attachment | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `Heur:Email.ArchivedExecutable`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test` | | mbox | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `Heur:Email.ExecutableAttachment`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test` | | PDF attachment (FlateDecode) | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `Heur:PDF.EmbeddedExecutable`, `Heur:PDF.EmbeddedFile`, `XProtect:EICAR` | | base64 file | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test`, `YARA-ext:BINARYALERT_Eicar_Substring_Test` | | Unix compress (.Z) | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test`, `YARA-ext:BINARYALERT_Eicar_Substring_Test` | | zstd | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test`, `YARA-ext:BINARYALERT_Eicar_Substring_Test` | | tar.zst | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test`, `YARA-ext:BINARYALERT_Eicar_Substring_Test` | | lz4 | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test`, `YARA-ext:BINARYALERT_Eicar_Substring_Test` | | 7z | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test`, `YARA-ext:BINARYALERT_Eicar_Substring_Test` | | 7z, password 'infected', names encrypted | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `Heur:Archive.Encrypted`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test` | | zip, password 'infected' (ZipCrypto) | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `Heur:Archive.Encrypted`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test` | | zip, password 'infected' (AES-256) | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `Heur:Archive.Encrypted`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test` | | zip, password a year (2024) | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `Heur:Archive.Encrypted`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test` | | xar | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test`, `YARA-ext:BINARYALERT_Eicar_Substring_Test` | | macOS .pkg (pkgbuild) | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test`, `YARA-ext:BINARYALERT_Eicar_Substring_Test` | | DMG UDZO | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test`, `YARA-ext:BINARYALERT_Eicar_Substring_Test` | | DMG UDBZ | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test`, `YARA-ext:BINARYALERT_Eicar_Substring_Test` | | DMG ULFO | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test`, `YARA-ext:BINARYALERT_Eicar_Substring_Test` | | DMG ULMO | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test`, `YARA-ext:BINARYALERT_Eicar_Substring_Test` | | DMG UDRO | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test`, `YARA-ext:BINARYALERT_Eicar_Substring_Test` | | ISO 9660 | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test`, `YARA-ext:BINARYALERT_Eicar_Substring_Test` | | zip inside DMG | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test`, `YARA-ext:BINARYALERT_Eicar_Substring_Test` | | app bundle (Contents/Resources) | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test`, `YARA-ext:BINARYALERT_Eicar_Substring_Test` | | DMG inside zip | malicious | MALICIOUS | Y | `ClamAV:Eicar-Test-Signature`, `XProtect:EICAR`, `YARA-ext:BINARYALERT_Eicar_Av_Test`, `YARA-ext:BINARYALERT_Eicar_Substring_Test` | ### Clean twins (near-miss files) The same 57 containers built without the EICAR string. Expected verdict: clean (no alarm). 57/57 passed: 53 were rated `CLEAN` and 4 `INFO`. The four `INFO` verdicts are the password-protected archives, which get the informational finding `Heur:Archive.Encrypted` (an encrypted archive cannot be inspected); `INFO` is below the alarm threshold, so it counts as a pass. | Test | Expected | Verdict | OK | Detecting engines (signature names) | |---|---|---|---|---| | plain .com | clean | CLEAN | Y | — | | plain + CRLF (AMTSO style) | clean | CLEAN | Y | — | | plain .txt | clean | CLEAN | Y | — | | plain, no extension | clean | CLEAN | Y | — | | zip (deflate) | clean | CLEAN | Y | — | | zip (stored) | clean | CLEAN | Y | — | | zip (bzip2) | clean | CLEAN | Y | — | | zip (lzma) | clean | CLEAN | Y | — | | zip in folder inside zip | clean | CLEAN | Y | — | | zip nested 2× | clean | CLEAN | Y | — | | zip nested 3× | clean | CLEAN | Y | — | | zip nested 5× | clean | CLEAN | Y | — | | zip nested 8× | clean | CLEAN | Y | — | | gzip | clean | CLEAN | Y | — | | bzip2 | clean | CLEAN | Y | — | | xz | clean | CLEAN | Y | — | | lzma (alone) | clean | CLEAN | Y | — | | tar | clean | CLEAN | Y | — | | tar.gz | clean | CLEAN | Y | — | | tar.bz2 | clean | CLEAN | Y | — | | tar.xz | clean | CLEAN | Y | — | | cpio (newc) | clean | CLEAN | Y | — | | ar | clean | CLEAN | Y | — | | deb (ar + tar.gz) | clean | CLEAN | Y | — | | jar | clean | CLEAN | Y | — | | docx (OOXML part) | clean | CLEAN | Y | — | | xlsx (OOXML part) | clean | CLEAN | Y | — | | odt (ODF) | clean | CLEAN | Y | — | | epub | clean | CLEAN | Y | — | | crx (Chrome extension) | clean | CLEAN | Y | — | | nupkg | clean | CLEAN | Y | — | | asar (Electron) | clean | CLEAN | Y | — | | e-mail attachment (base64) | clean | CLEAN | Y | — | | e-mail with zip attachment | clean | CLEAN | Y | — | | mbox | clean | CLEAN | Y | — | | PDF attachment (FlateDecode) | clean | CLEAN | Y | — | | base64 file | clean | CLEAN | Y | — | | Unix compress (.Z) | clean | CLEAN | Y | — | | zstd | clean | CLEAN | Y | — | | tar.zst | clean | CLEAN | Y | — | | lz4 | clean | CLEAN | Y | — | | 7z | clean | CLEAN | Y | — | | 7z, password 'infected', names encrypted | clean | INFO | Y | `Heur:Archive.Encrypted` | | zip, password 'infected' (ZipCrypto) | clean | INFO | Y | `Heur:Archive.Encrypted` | | zip, password 'infected' (AES-256) | clean | INFO | Y | `Heur:Archive.Encrypted` | | zip, password a year (2024) | clean | INFO | Y | `Heur:Archive.Encrypted` | | xar | clean | CLEAN | Y | — | | macOS .pkg (pkgbuild) | clean | CLEAN | Y | — | | DMG UDZO | clean | CLEAN | Y | — | | DMG UDBZ | clean | CLEAN | Y | — | | DMG ULFO | clean | CLEAN | Y | — | | DMG ULMO | clean | CLEAN | Y | — | | DMG UDRO | clean | CLEAN | Y | — | | ISO 9660 | clean | CLEAN | Y | — | | zip inside DMG | clean | CLEAN | Y | — | | app bundle (Contents/Resources) | clean | CLEAN | Y | — | | DMG inside zip | clean | CLEAN | Y | — | ## 3. SHA-256 hashes from Objective-See reports 2019–2024 33 SHA-256 hashes of Mac malware published in Objective-See's yearly Mac malware reports, checked against the scanner's hash databases only (no file is scanned). 8/33 (24.2%) are known; the verdict column shows the name of the matching database entry. By report year: 2019 0/2, 2020 0/1, 2022 6/20, 2023 1/3, 2024 1/7. | Sample (year, family / file) | SHA-256 | Expected | Verdict (database match) | OK | |---|---|---|---|---| | 2019 Siggen (HEYgiNb.py) | `f5808e9b9d204f646e33bbc4279b98b97b34086ffc3e9fb2ac828a8161099ee8` | known | none | N | | 2019 NewTab | `9237dc996a9e2c939f74414c87350009f11b147f1703823062f790921be30be9` | known | none | N | | 2020 watchcat (NukeSpeed) | `3bb96bfaf492782b38985f4bd6b7e7f9dc22c1332b42bb74b16041298fd31f93` | known | none | N | | 2022 SysJoker | `1a9a5c797777f37463b44de2b49a7f95abca786db3977dcdac0f79da739c08ac` | known | `Generic.macho` | Y | | 2022 SysJoker | `fe99db3268e058e1204aff679e0726dc77fd45d06757a5fda9eafc6a28cfb8df` | known | `Generic.macho` | Y | | 2022 SysJoker | `d0febda3a3d2d68b0374c26784198dc4309dbe4a8978e44bb7584fd832c325f0` | known | `Generic.macho` | Y | | 2022 DazzleSpy | `f9ad42a9bd9ade188e997845cae1b0587bf496a35c3bffacd20fefe07860a348` | known | none | N | | 2022 CoinMiner | `fe3700a52e86e250a9f38b7a5a48397196e7832fd848a7da3cc02fe52f49cdcf` | known | none | N | | 2022 CoinMiner | `fabe0b41fb5bce6bda8812197ffd74571fc9e8a5a51767bcceef37458e809c5c` | known | none | N | | 2022 CoinMiner | `a2909754783bb5c4fd6955bcebc356e9d6eda94f298ed3e66c7e13511275fbc4` | known | none | N | | 2022 Gimmick | `2a9296ac999e78f6c0bee8aca8bfa4d4638aa30d9c8ccc65124b1cbfc9caab5f` | known | `Generic.macho` | Y | | 2022 oRAT | `ee07dfd6443af8f20f5f11effb9cbcec07e125697a28aee78718caeed17f1407` | known | none | N | | 2022 Pymafka | `b117f042fe9bac7c7d39eab98891c2465ef45612f5355beea8d3c4ebd0665b45` | known | `Generic.macho` | Y | | 2022 VPN Trojan (Covid) | `7831806172857a563d7b4789acddc98fc11763aaf3cedf937630b4a9dce31419` | known | none | N | | 2022 VPN Trojan (Covid) | `d9bba1cfca6b1d20355ce08eda37d6d0bca8cb8141073b699000d05025510dcc` | known | none | N | | 2022 CloudMensis | `317ce26cae14dc9a5e4d4667f00fee771b4543e91c944580bbb136e7fe339427` | known | none | N | | 2022 CloudMensis | `b8a61adccefb13b7058e47edcd10a127c483403cf38f7ece126954e95e86f2bd` | known | none | N | | 2022 rShell | `3a9e72b3810b320fa6826a1273732fee7a8e2b2e5c0fd95b8c36bbab970e830a` | known | none | N | | 2022 rShell | `8c3be245cbbe9206a5d146017c14b8f965ab7045268033d70811d5bcc4b796ec` | known | none | N | | 2022 Insekt (exploit) | `ef130f1941077ffe383fe90e241620dde771cd0dd496dad29d2048d5fc478faf` | known | `Generic.macho` | Y | | 2022 Insekt (payload.so) | `ae9f370c89f0191492ed9c17a224d9c41778b47ca2768f732b4de6ee7d0d1459` | known | none | N | | 2022 KeySteal (ResignTool) | `410da3923ea30d5fdd69b9ae69716b094d276cc609f76590369ff254f71c65da` | known | none | N | | 2022 KeySteal | `f5b4a388fee4183dfa46908000c5c50dceb4bf8025c4cfcb4d478c5d03833202` | known | none | N | | 2023 LockBit macOS | `3e4bbd21756ae30c24ff7d6942656be024139f8180b7bddd4e5c62a9dfbd8c79` | known | `Generic.macho` | Y | | 2023 GoSorry (archive) | `c80c8c0e961e1692c5afdb8b3dd48e84fb6bb24911ad449ef75976cbf5bf1199` | known | none | N | | 2023 GoSorry (liboptimizer.dylib) | `17b84ed0f8bdbb6619d64bf686a6b990f497b378e3b70792e2cd896c2abda14d` | known | none | N | | 2024 fake MiroTalk.dmg | `9abf6b93eafb797a3556bea1fe8a3b7311d2864d5a9a3687fce84bc1ec4a428c` | known | `Generic.dmg` | Y | | 2024 trojanised ultraedit.dmg | `9eb7bda5ffbb1a7549b1e481b1a6ed6efe2e28d0463370c87630fed74eee6228` | known | none | N | | 2024 libConfigurer64.dylib (ultraedit) | `ce40829673687b48d68defa3176c8ab59a2a50ee9c658fe46a5de7692fbc112d` | known | none | N | | 2024 .fseventsd (ultraedit) | `1b2d50cdacfd39205c3caff2925eb35b59312dbe099bd3a98ae3b2f2f909ab17` | known | none | N | | 2024 Khepri build (.test) | `21457a89317e4c6b8aaee5e461a6b4e444c736243d550efd0e681eda05b97007` | known | none | N | | 2024 vShell (Cloudflare Security Authenticator.dmg) | `c5686b85efb3ebf2ce07dba4192195c3dac7c335a371b7bcfbf52d5fb15cb507` | known | none | N | | 2024 SNOWLIGHT-like | `d1b7f3d88ed8d37774b229cc46df2c08c95067736cf418b4a76a8403cae4e2a6` | known | none | N | ## 4. Mac malware families 2019–2025 with a family-named rule 83 Mac malware families from 2019–2025. A family passes if at least one signature source has a rule or signature named after that family (expected: `named signature`). 57/83 (68.7%) pass. The "Sources" column shows where the named signature exists: `YARA` (YARA rules), `ClamAV` (ClamAV signatures), `hash` (hash database entry with the family name). For 2024 SnowLight only a hash entry exists (`Hash only` = Y), which does not count as a pass. | Year | Families with a named rule | Total | Rate | |---|---|---|---| | 2019 | 6 | 8 | 75.0% | | 2020 | 5 | 7 | 71.4% | | 2021 | 7 | 8 | 87.5% | | 2022 | 8 | 12 | 66.7% | | 2023 | 15 | 18 | 83.3% | | 2024 | 11 | 18 | 61.1% | | 2025 | 5 | 12 | 41.7% | | All | 57 | 83 | 68.7% | Signature sources among the 57 passing families: YARA only 31, YARA+ClamAV 17, YARA+ClamAV+hash 5, YARA+hash 3, ClamAV only 1. | Family (year) | Expected | Sources with a family-named signature | Hash only | OK | |---|---|---|---|---| | 2019 BirdMiner / LoudMiner | named signature | ClamAV | N | Y | | 2019 CookieMiner | named signature | none | N | N | | 2019 GMERA | named signature | YARA+ClamAV | N | Y | | 2019 Mokes | named signature | YARA+ClamAV | N | Y | | 2019 NetWire | named signature | YARA+ClamAV | N | Y | | 2019 Siggen | named signature | none | N | N | | 2019 Yort | named signature | YARA | N | Y | | 2019 Lazarus MacLoader / AppleJeus | named signature | YARA+ClamAV | N | Y | | 2020 GravityRAT | named signature | none | N | N | | 2020 IPStorm | named signature | YARA | N | Y | | 2020 Dacls | named signature | YARA | N | Y | | 2020 EvilQuest / ThiefQuest | named signature | YARA+ClamAV | N | Y | | 2020 FinSpy | named signature | YARA+ClamAV | N | Y | | 2020 WatchCat | named signature | none | N | N | | 2020 XCSSET | named signature | YARA+ClamAV | N | Y | | 2021 ElectroRAT | named signature | YARA+ClamAV | N | Y | | 2021 ElectrumStealer | named signature | none | N | N | | 2021 Macma / CDDS | named signature | YARA+ClamAV | N | Y | | 2021 Silver Sparrow | named signature | YARA+ClamAV | N | Y | | 2021 XcodeSpy / EggShell | named signature | YARA+ClamAV | N | Y | | 2021 ZuRu | named signature | YARA | N | Y | | 2021 WildPressure (Milum) | named signature | YARA+ClamAV+hash | N | Y | | 2021 XLoader | named signature | YARA | N | Y | | 2022 CloudMensis | named signature | YARA+hash | N | Y | | 2022 CrateDepression | named signature | none | N | N | | 2022 DazzleSpy | named signature | YARA+hash | N | Y | | 2022 Gimmick | named signature | YARA+ClamAV | N | Y | | 2022 Insekt | named signature | none | N | N | | 2022 KeySteal | named signature | YARA | N | Y | | 2022 oRAT | named signature | YARA | N | Y | | 2022 Pymafka | named signature | none | N | N | | 2022 rShell | named signature | YARA | N | Y | | 2022 SentinelSneak | named signature | none | N | N | | 2022 SysJoker | named signature | YARA+ClamAV+hash | N | Y | | 2022 VPN Trojan (Covid) | named signature | YARA | N | Y | | 2023 SmoothOperator (3CX) | named signature | YARA | N | Y | | 2023 Atomic Stealer (AMOS) | named signature | YARA+ClamAV+hash | N | Y | | 2023 Geacon | named signature | YARA | N | Y | | 2023 GoSorry | named signature | none | N | N | | 2023 JaskaGO | named signature | YARA | N | Y | | 2023 JokerSpy | named signature | YARA | N | Y | | 2023 KandyKorn | named signature | YARA+ClamAV | N | Y | | 2023 LockBit (macOS) | named signature | YARA | N | Y | | 2023 MacStealer | named signature | YARA+ClamAV | N | Y | | 2023 MetaStealer | named signature | none | N | N | | 2023 NokNok | named signature | YARA | N | Y | | 2023 ObjCShellz | named signature | YARA | N | Y | | 2023 PureLand | named signature | YARA | N | Y | | 2023 Realst | named signature | YARA+ClamAV | N | Y | | 2023 RustBucket | named signature | YARA | N | Y | | 2023 SparkRAT | named signature | YARA | N | Y | | 2023 Turtle | named signature | YARA+ClamAV | N | Y | | 2023 WSClient | named signature | none | N | N | | 2024 Activator | named signature | none | N | N | | 2024 Banshee | named signature | YARA+ClamAV | N | Y | | 2024 BeaverTail | named signature | YARA+ClamAV+hash | N | Y | | 2024 CloudChat | named signature | none | N | N | | 2024 Cthulhu | named signature | YARA | N | Y | | 2024 Hidden Risk | named signature | YARA | N | Y | | 2024 HZ Rat | named signature | YARA | N | Y | | 2024 InletDrift | named signature | none | N | N | | 2024 LightSpy | named signature | YARA+ClamAV+hash | N | Y | | 2024 NotLockBit | named signature | none | N | N | | 2024 Poseidon | named signature | YARA | N | Y | | 2024 PyStealer | named signature | none | N | N | | 2024 RustDoor | named signature | YARA | N | Y | | 2024 RustyAttr | named signature | none | N | N | | 2024 SnowLight | named signature | hash | Y | N | | 2024 SpectralBlur | named signature | YARA | N | Y | | 2024 ToDoSwift | named signature | YARA | N | Y | | 2024 VShell | named signature | YARA | N | Y | | 2025 ChillyHell | named signature | YARA | N | Y | | 2025 DigitStealer | named signature | none | N | N | | 2025 FlexibleFerret | named signature | YARA | N | Y | | 2025 Frigid Stealer | named signature | none | N | N | | 2025 Kitty Stealer | named signature | none | N | N | | 2025 Koi Stealer | named signature | none | N | N | | 2025 MacSync | named signature | YARA+hash | N | Y | | 2025 NightPaw | named signature | none | N | N | | 2025 Paradox | named signature | YARA | N | Y | | 2025 PasivRobber | named signature | YARA | N | Y | | 2025 Phexia | named signature | none | N | N | | 2025 RN Loader / RN Stealer | named signature | none | N | N | --- # Malware detection: test samples vs held-out clean files > The full engine detects 519/528 test samples at INFO+, 479/528 at SUSPICIOUS+ and 318/528 at MALICIOUS, with 0 SUSPICIOUS+ false alarms on clean files never used for tuning (0/2973, later 0/2812); a fix of two context heuristics cut held-out INFO+ findings from 119/2973 (4.00%) to 10/2973 (0.34%). Area: Security (macOS malware scanning). Page: https://benchmarks.site/malware-detection-held-out/. Methodology and caveats: https://benchmarks.site/methodology/#malware The full engine (all databases + heuristics) scans: - 528 malicious test samples (expected: detected), - 14 clean test samples (expected: no alarm), - clean files from a real Mac that were used while tuning the heuristics, - held-out clean files that were **never** used for tuning — the main false-alarm measure. A false alarm is a clean file that reaches the given threshold. Thresholds are cumulative: INFO+ = `INFO`, `SUSPICIOUS`, `PUA` or `MALICIOUS`; SUSPICIOUS+ = `SUSPICIOUS`, `PUA` or `MALICIOUS`; MALICIOUS = `MALICIOUS` only. Only SUSPICIOUS+ is shown to the user as an alarm. ## Runs | Run | Date (run end) | Files scanned | Malicious test samples | Clean test samples | Clean (tuning) | Clean (held-out) | Scan time (s) | |---|---|---|---|---|---|---|---| | `20261103` | 2026-11-03 13:43 | 5824 | 528 | 14 | 2309 | 2973 | 462 | | `20261103-fix` | 2026-11-03 14:01 | 5824 | 528 | 14 | 2309 | 2973 | 827 | | `20261109-speed` | 2026-11-09 13:38 | 5431 | 528 | 14 | 2077 | 2812 | 3558 | Scan time is the elapsed time of the full scan as printed in the run log. The runs were not made under controlled load, so the times are not comparable as a speed benchmark (see [Performance](/scanner-performance/)). The `20261109-speed` run scanned a smaller clean set (393 fewer files: 232 fewer in the tuning set and 161 fewer in the held-out set). ## 1. Results by threshold ### Run `20261103` | Threshold | Detection (test samples) | False alarms: held-out clean (never tuned) | False alarms: clean used for tuning | False alarms: clean test samples | |---|---|---|---|---| | INFO+ | 519/528 (98.30%) | 119/2973 (4.00%) | 32/2309 (1.39%) | 1/14 (7.14%) | | SUSPICIOUS+ | 479/528 (90.72%) | 0/2973 (0.00%) | 0/2309 (0.00%) | 0/14 (0.00%) | | MALICIOUS | 318/528 (60.23%) | 0/2973 (0.00%) | 0/2309 (0.00%) | 0/14 (0.00%) | ### Run `20261103-fix` | Threshold | Detection (test samples) | False alarms: held-out clean (never tuned) | False alarms: clean used for tuning | False alarms: clean test samples | |---|---|---|---|---| | INFO+ | 519/528 (98.30%) | 10/2973 (0.34%) | 4/2309 (0.17%) | 1/14 (7.14%) | | SUSPICIOUS+ | 479/528 (90.72%) | 0/2973 (0.00%) | 0/2309 (0.00%) | 0/14 (0.00%) | | MALICIOUS | 318/528 (60.23%) | 0/2973 (0.00%) | 0/2309 (0.00%) | 0/14 (0.00%) | ### Run `20261109-speed` | Threshold | Detection (test samples) | False alarms: held-out clean (never tuned) | False alarms: clean used for tuning | False alarms: clean test samples | |---|---|---|---|---| | INFO+ | 519/528 (98.30%) | 9/2812 (0.32%) | 4/2077 (0.19%) | 1/14 (7.14%) | | SUSPICIOUS+ | 479/528 (90.72%) | 0/2812 (0.00%) | 0/2077 (0.00%) | 0/14 (0.00%) | | MALICIOUS | 318/528 (60.23%) | 0/2812 (0.00%) | 0/2077 (0.00%) | 0/14 (0.00%) | Verdict counts on the 528 test samples were the same in all three runs: 318 `MALICIOUS`, 141 `SUSPICIOUS`, 20 `PUA`, 40 `INFO`, 9 not detected. ## 2. Held-out clean files by category (INFO+ findings) No held-out file reached SUSPICIOUS+ or MALICIOUS in any run, so only INFO+ counts are shown. | Category | Description | `20261103` | `20261103-fix` | `20261109-speed` | |---|---|---|---|---| | `app_frameworks_macho` | Mach-O binaries inside application frameworks | 0/725 (0.00%) | 0/725 (0.00%) | 0/641 (0.00%) | | `app_resources_scripts` | Scripts from application resource folders | 3/500 (0.60%) | 3/500 (0.60%) | 2/464 (0.43%) | | `homebrew_cellar` | Files from installed Homebrew packages | 2/800 (0.25%) | 2/800 (0.25%) | 2/800 (0.25%) | | `python_libs` | Installed Python library files | 0/400 (0.00%) | 0/400 (0.00%) | 0/400 (0.00%) | | `user_documents` | User document files | 0/22 (0.00%) | 0/22 (0.00%) | 0/22 (0.00%) | | `installers_archives` | Downloaded installers and archives | 5/30 (16.67%) | 2/30 (6.67%) | 2/30 (6.67%) | | `app_support` | Application support data | 109/492 (22.15%) | 3/492 (0.61%) | 3/451 (0.67%) | | `project_deps` | Dependencies inside software projects | 0/4 (0.00%) | 0/4 (0.00%) | 0/4 (0.00%) | | All held-out | | 119/2973 (4.00%) | 10/2973 (0.34%) | 9/2812 (0.32%) | ## 3. What changed between runs - `20261103` → `20261103-fix`: almost all held-out INFO+ findings came from `app_support` (109/492) and `installers_archives` (5/30). Across all 5824 scanned files, the number of `INFO` files with the context heuristic `Heur:Context.DownloadedUnsigned` fell from 138 to 2, and `Heur:Context.Name.Random` (32 `INFO` files) no longer appears. Total `INFO` verdicts fell from 192 to 55. Detection at every threshold stayed the same. - `20261103-fix` → `20261109-speed`: same detection and the same per-threshold false-alarm results on a slightly smaller clean set; held-out INFO+ 9/2812 (0.32%). ## 4. Observations - At the alarm threshold (SUSPICIOUS+), detection is 479/528 (90.72%) with 0 false alarms on held-out clean files in all three runs. - `MALICIOUS` alone catches 318/528 (60.23%); about a third of detected samples are rated only `SUSPICIOUS` or `PUA`. - 9 of the 528 test samples are not detected at any level. - The single clean test sample with an INFO+ finding (1/14) is below the alarm threshold. - Small categories (`project_deps` 4, `user_documents` 22, `installers_archives` 30) give rates with large uncertainty; one file in `installers_archives` is 3.33 percentage points. --- # Malware scanning: static-heuristic false alarms before and after fixes > Fixes to the static heuristics removed all 6 SUSPICIOUS false alarms on 2945 clean Mac files (0.20% → 0/2944) while raising heuristic-only detection on the 528 test samples from 107 to 135 at SUSPICIOUS+; an intermediate change that briefly added 2 new false alarms was reverted. Area: Security (macOS malware scanning). Page: https://benchmarks.site/malware-static-false-positives/. Methodology and caveats: https://benchmarks.site/methodology/#malware Dates: 2026-11-02 and 2026-11-03. Two measurements are taken for each version of the engine: - **Clean files:** about 2900 clean files from a real Mac, grouped into 12 categories. Any `SUSPICIOUS`, `PUA` or `MALICIOUS` verdict is a false alarm ("flagged"); `INFO` findings are counted separately. - **Fixtures:** 528 malicious test samples, 14 clean test samples and 48 neutral archive files (no expected verdict), scanned in two modes: `heur` (static heuristics only) and `yara` (heuristics together with YARA rules). Runs: | Run | Date | Description | |---|---|---| | `before` | 2026-11-02 | Heuristics before the fixes. | | `after` | 2026-11-02 | After the fixes, including an extraction of embedded scripts from Mach-O binaries. | | `after-final` | 2026-11-02 | The `after` run with the records it flagged re-scanned after reverting the Mach-O embedded-script extraction. Removing an analyzer path can only remove detections; the two later script rules were checked separately: 0 matches on all 2380 clean text files. | | `bench-20261103` | 2026-11-03 | Re-measurement the next day, on the day of the first [fair benchmark](/malware-detection-held-out/) run. | ## 1. Clean files: summary | Run | Files | False alarms (SUSPICIOUS+) | Of which `MALICIOUS` | `INFO` findings | Summed scan time (s) | |---|---|---|---|---|---| | `before` | 2945 | 6 (0.20%) | 0 | 61 (2.07%) | 4149.2 | | `after` | 2944 | 2 (0.07%) | 1 | 61 (2.07%) | 4954.6 | | `after-final` | 2944 | 0 (0.00%) | 0 | 62 (2.11%) | 4954.6 (from `after`) | | `bench-20261103` | 2910 | 0 (0.00%) | 0 | 46 (1.58%) | 1044.6 | Scan time is the sum of per-file scan times. ## 2. Clean files by category Each cell: false alarms (SUSPICIOUS+) / `INFO` findings / files. | Category | Description | `before` | `after` | `after-final` | `bench-20261103` | |---|---|---|---|---|---| | `apps_plist` | `Info.plist` files of installed apps | 0 / 0 / 36 | 0 / 0 / 36 | 0 / 0 / 36 | 0 / 0 / 36 | | `apps_helper` | Helper executables inside app bundles (Mach-O) | 4 / 0 / 90 | 0 / 0 / 90 | 0 / 0 / 90 | 0 / 0 / 90 | | `apps_main` | Main executables of installed apps | 1 / 2 / 41 | 1 / 1 / 41 | 0 / 2 / 41 | 0 / 2 / 41 | | `apps_script` | Scripts inside app bundles | 1 / 0 / 84 | 0 / 1 / 84 | 0 / 1 / 84 | 0 / 1 / 84 | | `homebrew_bin` | Executables installed by Homebrew | 0 / 0 / 495 | 1 / 0 / 495 | 0 / 0 / 495 | 0 / 1 / 495 | | `extra_script` | Additional script files (JavaScript, Python, shell, text, PowerShell, Perl) | 0 / 46 / 477 | 0 / 46 / 477 | 0 / 46 / 477 | 0 / 29 / 459 | | `launchd_plist` | launchd job definitions (plist) | 0 / 0 / 131 | 0 / 0 / 131 | 0 / 0 / 131 | 0 / 0 / 131 | | `sysapps_plist` | `Info.plist` files of macOS system apps | 0 / 0 / 65 | 0 / 0 / 65 | 0 / 0 / 65 | 0 / 0 / 65 | | `sysapps_main` | Main executables of macOS system apps | 0 / 0 / 66 | 0 / 0 / 66 | 0 / 0 / 66 | 0 / 0 / 66 | | `docs_own` | Source and config files of the author's own projects | 0 / 7 / 660 | 0 / 7 / 659 | 0 / 7 / 659 | 0 / 7 / 643 | | `docs_deps` | Dependency files inside those projects | 0 / 6 / 549 | 0 / 6 / 549 | 0 / 6 / 549 | 0 / 6 / 549 | | `usr_bin_script` | macOS system command-line scripts (Perl, shell, Ruby) | 0 / 0 / 251 | 0 / 0 / 251 | 0 / 0 / 251 | 0 / 0 / 251 | | All | | 6 / 61 / 2945 | 2 / 61 / 2944 | 0 / 62 / 2944 | 0 / 46 / 2910 | ## 3. Which detections caused the false alarms ### `before` (6 false alarms, all `SUSPICIOUS`) | Detection | Severity | Files | Categories | |---|---|---|---| | `Heur:MachO.CertificateRevoked` | HIGH | 5 | `apps_helper` 4, `apps_main` 1 (together with `Heur:Context.DownloadedUnsigned`) | | `Heur:SHELL.RemoveQuarantine` + `Heur:SHELL.KeychainAccess` | MEDIUM + MEDIUM | 1 | `apps_script` 1 | ### `after` (2 false alarms) Both were caused by the new extraction of embedded scripts from Mach-O binaries: shell-script rules fired on strings inside two clean executables. | Category | Verdict | Score | Detections | |---|---|---|---| | `apps_main` | `MALICIOUS` | 231 | `Own:OWN_Script_Bash_CurlPipeShell_RawIpv4`, `Heur:SHELL.PipeToShell`, `Heur:SHELL.DecodePipe`, `Heur:SHELL.KeychainExfil`, `Heur:SHELL.DownloadExecute`, `Heur:SHELL.Exfil`, `Heur:MachO.Downloader`, `Heur:MachO.AdhocSigned` | | `homebrew_bin` | `SUSPICIOUS` | 53 | `Heur:SHELL.ReverseShellDevTcp`, `Heur:MachO.Downloader`, `Heur:MachO.AdhocSigned` | ### `after-final` The extraction was reverted and the two records re-scanned: the `apps_main` file became `INFO` (score 13) and the `homebrew_bin` file `CLEAN` (score 8). No false alarms remain. ### `INFO` findings on clean files `INFO` is below the alarm threshold. In `before`, 58 of the 61 `INFO` files carry `Heur:Context.DownloadedUnsigned` (45 in `extra_script`, 7 in `docs_own`, 6 in `docs_deps`) and 5 carry `Heur:Context.Name.Random` (all `extra_script`). In `bench-20261103`, `Heur:Context.DownloadedUnsigned` appears on 41 files and `Heur:Context.Name.Random` on 4. ## 4. Fixtures: detection on test samples The 14 clean test samples were never flagged (0/14) in any run and mode. | Run | Mode | INFO+ | SUSPICIOUS+ | MALICIOUS | |---|---|---|---|---| | `before` | `heur` | 168/528 (31.82%) | 107/528 (20.27%) | 31/528 (5.87%) | | `after` | `heur` | 197/528 (37.31%) | 135/528 (25.57%) | 43/528 (8.14%) | | `bench-20261103` | `heur` | 197/528 (37.31%) | 135/528 (25.57%) | 43/528 (8.14%) | | `before` | `yara` | 519/528 (98.30%) | 466/528 (88.26%) | 296/528 (56.06%) | | `after` | `yara` | 519/528 (98.30%) | 479/528 (90.72%) | 319/528 (60.42%) | | `bench-20261103` | `yara` | 519/528 (98.30%) | 479/528 (90.72%) | 319/528 (60.42%) | ### By sample group, `heur` mode (INFO+ / SUSPICIOUS+ / MALICIOUS) | Group | Samples | `before` | `after` | `bench-20261103` | |---|---|---|---|---| | `documents` | 15 | 15 / 15 / 8 | 15 / 15 / 8 | 15 / 15 / 8 | | `eicar` | 2 | 0 / 0 / 0 | 0 / 0 / 0 | 0 / 0 / 0 | | `scripts` | 16 | 16 / 16 / 5 | 16 / 16 / 6 | 16 / 16 / 6 | | `yara/gen` | 107 | 37 / 28 / 7 | 55 / 45 / 14 | 55 / 45 / 14 | | `yara/macos` | 60 | 33 / 17 / 5 | 41 / 26 / 8 | 41 / 26 / 8 | | `yara/native` | 16 | 4 / 3 / 0 | 4 / 3 / 0 | 4 / 3 / 0 | | `yara/scripts_web` | 312 | 63 / 28 / 6 | 66 / 30 / 7 | 66 / 30 / 7 | ### By sample group, `yara` mode (INFO+ / SUSPICIOUS+ / MALICIOUS) | Group | Samples | `before` | `after` | `bench-20261103` | |---|---|---|---|---| | `documents` | 15 | 15 / 15 / 10 | 15 / 15 / 10 | 15 / 15 / 10 | | `eicar` | 2 | 2 / 2 / 2 | 2 / 2 / 2 | 2 / 2 / 2 | | `scripts` | 16 | 16 / 16 / 9 | 16 / 16 / 9 | 16 / 16 / 9 | | `yara/gen` | 107 | 107 / 55 / 21 | 107 / 68 / 35 | 107 / 68 / 35 | | `yara/macos` | 60 | 60 / 60 / 54 | 60 / 60 / 60 | 60 / 60 / 60 | | `yara/native` | 16 | 7 / 6 / 4 | 7 / 6 / 4 | 7 / 6 / 4 | | `yara/scripts_web` | 312 | 312 / 312 / 196 | 312 / 312 / 199 | 312 / 312 / 199 | ### Neutral archives (48 files, no expected verdict) | Run | Mode | `CLEAN` | `INFO` | `SUSPICIOUS` | |---|---|---|---|---| | `before` | `heur` | 30 | 10 | 8 | | `before` | `yara` | 30 | 8 | 10 | | `after` | `heur` | 30 | 10 | 8 | | `after` | `yara` | 30 | 8 | 10 | | `bench-20261103` | `heur` | 30 | 10 | 8 | | `bench-20261103` | `yara` | 30 | 8 | 10 | ## 5. Observations - The fixes removed the `Heur:MachO.CertificateRevoked` false alarms on clean app helpers and main executables (5 files) and the script false alarm in `apps_script`; none of these files is flagged afterwards. - At the same time, heuristic-only detection on the test samples rose (SUSPICIOUS+ 107 → 135, MALICIOUS 31 → 43), and with YARA rules SUSPICIOUS+ rose from 466 to 479 and MALICIOUS from 296 to 319. - Extracting embedded scripts from Mach-O binaries produced 2 new false alarms on clean files (one rated `MALICIOUS`) and was reverted. - The `bench-20261103` clean set is 34 files smaller (2910 vs 2944), entirely in `extra_script` (459 vs 477) and `docs_own` (643 vs 659). --- # ML classifiers on TLSH fingerprints of Mach-O files > Gradient-boosted trees are the best model on TLSH features — GBDT 600×d6 reaches out-of-fold AUC 0.966 and 39.3% recall at a 0.1% false-positive rate, ahead of MLPs (18–24%) and logistic regression (9.5%); nearest-neighbour TLSH distance rules alone are far weaker (AUC 0.582). Area: Security (macOS malware scanning). Page: https://benchmarks.site/ml-tlsh-macho-classifiers/. Methodology and caveats: https://benchmarks.site/methodology/#malware Date: 2026-11-03. (file types `macho` and `macho_fat`). The question: can a classifier trained on TLSH fuzzy-hash fingerprints of Mach-O binaries recognise malware it has not seen, better than the plain rule "TLSH distance to a known sample ≤ threshold"? ## Dataset | Set | Count | |---|---| | Malicious Mach-O files (positives) | 1436 | | Clean Mach-O files (negatives) | 4353 | | Benchmark samples (`bench`) | 547 | | Malware families | 23 | Validation: grouped 5-fold cross-validation, where groups keep whole malware families and whole applications in one fold, so the test fold always contains families and apps not seen in training. Per fold this gives on average 287.2 positives and 870.6 negatives in the test part. ## Metrics - **AUC test** — ROC AUC on the test folds, averaged over the 5 folds (`auc_te`). - **OOF AUC** — ROC AUC on the pooled out-of-fold predictions of all 5 folds (each file scored by the model that did not see it in training) (`oof_auc`). - **Recall at 0.1% FP (OOF)** — share of the 1436 malicious files detected when the decision threshold on the pooled out-of-fold scores is set so that at most 0.1% of the 4353 clean files are flagged (`oof_rec_at_0.1`). This is the main number: a scanner must keep false alarms very low. - **FP at that threshold** — clean files flagged at that threshold (`oof_fp_at_0.1`); 4/4353 = 0.092% for every model. - **Precision (OOF)** — share of flagged files that are malicious at that threshold (`oof_prec_at_0.1`). - **Params** — number of trainable parameters; for GBDT the node count of full trees, trees × (2^(depth+1) − 1), e.g. 600 × 127 = 76200. - **Training seconds** — training time as recorded by the script (`seconds`). ## 1. Main results (out-of-fold, threshold at 0.1% FP) | Model | AUC test | OOF AUC | Recall at 0.1% FP | FP at threshold | Precision | Params | Training (s) | |---|---|---|---|---|---|---|---| | GBDT 600× d6 (600 trees, depth 6) | 0.967 | 0.966 | 39.3% (564/1436) | 4/4353 | 0.993 | 76200 | 568.2 | | GBDT 300× d4 (300 trees, depth 4) | 0.963 | 0.962 | 36.5% (524/1436) | 4/4353 | 0.992 | 9300 | 159.4 | | MLP 1M (561-1024-512-1) | 0.943 | 0.923 | 23.8% (342/1436) | 4/4353 | 0.988 | 1100801 | 307.0 | | MLP 0.2M (561-256-256-1) | 0.942 | 0.925 | 18.0% (259/1436) | 4/4353 | 0.985 | 209921 | 67.6 | | Logistic regression (linear) | 0.896 | 0.894 | 9.5% (137/1436) | 4/4353 | 0.972 | 562 | 1.9 | The input layer of the MLPs (561) is the TLSH feature vector length; logistic regression has 561 weights + 1 bias = 562 parameters. ## 2. Per-fold thresholds (summed over the 5 test folds) Here each test fold is evaluated with its own threshold (as set by the script) instead of one threshold on the pooled out-of-fold scores; counts are summed over the 5 folds. | Model | Recall test (mean over folds) | Malicious detected (sum) | False positives (sum) | Precision | AUC train | Recall train | |---|---|---|---|---|---|---| | GBDT 600× d6 | 36.0% | 498/1436 | 2/4353 | 0.996 | 1.000 | 100.0% | | GBDT 300× d4 | 32.8% | 479/1436 | 2/4353 | 0.996 | 1.000 | 100.0% | | MLP 1M (561-1024-512-1) | 21.4% | 378/1436 | 3/4353 | 0.992 | 1.000 | 100.0% | | MLP 0.2M (561-256-256-1) | 21.7% | 296/1436 | 2/4353 | 0.993 | 1.000 | 100.0% | | Logistic regression (linear) | 4.7% | 72/1436 | 5/4353 | 0.935 | 0.992 | 50.7% | Mean false positives per fold: GBDT 0.4, MLP 1M 0.6, MLP 0.2M 0.4, logistic regression 1.0 (out of 870.6 clean files per fold). ## 3. Baseline: TLSH nearest-neighbour rule (no model) A file is flagged if its TLSH distance to the nearest known malicious sample is ≤ the threshold. | Rule | Recall | False positives | Precision | AUC test | Params | |---|---|---|---|---|---| | TLSH-NN ≤ 30 | 4.7% (68/1436) | 1/4353 (0.023%) | 0.986 | 0.582 | 0 | | TLSH-NN ≤ 50 | 13.7% (197/1436) | 7/4353 (0.161%) | 0.966 | 0.582 | 0 | The ≤ 50 rule already exceeds the 0.1% FP budget (7/4353 = 0.161%); the project summary lists the TLSH ≤ 50 rule alone as giving ~0% detection at 0.1% FP. ## 4. Additional recall figures recorded by the script The results file also contains two further recall values at the 0.1% FP threshold. Their exact definitions are in the training script, not in the results file; the field names are given verbatim. | Model | `oof_far_recall` | `oof_union_recall` | |---|---|---| | GBDT 600× d6 | 30.9% (383/1239) | 40.4% (580/1436) | | GBDT 300× d4 | 28.7% (355/1239) | 38.4% (552/1436) | | MLP 1M (561-1024-512-1) | 19.8% (245/1239) | 30.8% (442/1436) | | MLP 0.2M (561-256-256-1) | 12.0% (149/1239) | 24.1% (346/1436) | | Logistic regression (linear) | 9.2% (114/1239) | 21.7% (311/1436) | `oof_far_recall` is computed over a subset of 1239 of the 1436 malicious files; `oof_union_recall` is computed over all 1436. ## 5. Observations - Trees win clearly: GBDT 600×d6 detects 39.3% of unseen-family malware at 0.1% FP, about 4× logistic regression (9.5%) and 1.65× the larger MLP (23.8%). - The larger GBDT gains 2.8 percentage points of recall over GBDT 300×d4 at 8× the parameters and 3.6× the training time. - All neural and tree models fit the training folds perfectly (train AUC 1.000, train recall 100%), so the test numbers measure generalisation to new families and apps. - The MLP 1M has a higher per-fold test AUC than the MLP 0.2M (0.943 vs 0.942) but a lower OOF AUC (0.923 vs 0.925); its recall at 0.1% FP is higher (23.8% vs 18.0%). - A TLSH distance rule alone is weak on unseen families (AUC 0.582); the learned models are needed for generalisation. --- # Scanner performance: database loading, re-scans, TLSH in SQL, Aho-Corasick > Database load time dropped from 60–77 s to ~11 s, a re-scan of Numbers.app (33819 files) from 131.7 s to 24.7 s, and moving TLSH candidate selection into SQL gave identical results (718/718) ~1.6× faster; `ahocorasick_rs` built lazily beats loading a pickled `pyahocorasick` automaton, and threaded sharding gives no speedup because of the GIL. Area: Security (macOS malware scanning). Page: https://benchmarks.site/scanner-performance/. Methodology and caveats: https://benchmarks.site/methodology/#malware These are manual measurements taken on the development Mac in November 2026. There are no raw result files for them; the numbers below are the recorded results. ## Ours (own engine) | Measurement | Date | Before | After | Result | |---|---|---|---|---| | Database load time | 2026-11-09 | 60–77 s | ~11 s | about 5.5–7× faster | | Scan memory: re-scan of Numbers.app (33819 files) | 2026-11-11 | 131.7 s | 24.7 s | about 5.3× faster | | TLSH candidate restriction in SQL (`tlsh_candidates_within`) | 2026-11-09 | — | — | 718/718 identical results, ~1.6× faster | - **Database load time:** time to load the signature and hash databases at engine start. - **Scan memory:** the scanner remembers files it has already scanned, so a second scan of the same application bundle (Numbers.app, 33819 files) only has to handle what changed. - **TLSH in SQL:** the set of TLSH fuzzy-hash candidates within a distance limit is restricted already in the SQL query (`tlsh_candidates_within`); all 718 compared results were identical to the previous method. ## External libraries | Library / approach | Date | Measurement | Result | |---|---|---|---| | `pyahocorasick` (save/load of a prebuilt automaton, pickle) | 2026-11-09 | load time | 76 s | | `ahocorasick_rs` (build the automaton) | 2026-11-09 | build time | ~25 s | | Building the automaton in threads (sharding) | 2026-11-09 | build time | no speedup (GIL) | - Loading a pickled `pyahocorasick` automaton (76 s) was slower than building one from scratch with `ahocorasick_rs` (~25 s), so `ahocorasick_rs` stays, and the automaton is built lazily (only when it is first needed). - Splitting the pattern set into shards and building them in parallel threads gave no speedup, because of Python's global interpreter lock (GIL). --- # Public AI benchmarks — results collected on benchmarks.site 52 public benchmarks and leaderboards; results copied from each publisher with the source stated (retrieved 2026-11-17 unless noted). Page: https://benchmarks.site/internet/ ## General knowledge & reasoning - [ARC-AGI (ARC Prize)](https://benchmarks.site/bots/internet/arc-agi-arc-prize.md) — Puzzles that most people find easy but that require spotting a brand-new pattern rather than recalling facts; the newest version (ARC-AGI-3) drops the AI into small interactive games it has never seen. It is a popular test of whether AI can genuinely learn new skills on the fly. Leader: 1 / GPT-6 Astra - Provider Adapter (High) / OpenAI / 99.9 / 18816.63. - [CritPt](https://benchmarks.site/bots/internet/critpt.md) — Unpublished research-level physics problems written by working physicists, in areas like quantum physics and astrophysics. It tests whether AI can do the kind of reasoning needed in real scientific research. Leader: 1 / Gemini 4 Carbon (High) / Google DeepMind / 36.0 / 2026-11-10. - [EQ-Bench](https://benchmarks.site/bots/internet/eq-bench.md) — Tests how well AI models handle emotionally sensitive conversations with simulated people who have different personalities and needs. Leader: 1 / claude-opus-5 / 1364.9 / 1349.5 - 1378.4. - [Humanity's Last Exam (HLE)](https://benchmarks.site/bots/internet/humanitys-last-exam-hle.md) — About 2,500 extremely hard questions written by experts in over 100 subjects, from classics to quantum physics, meant to sit at the edge of human knowledge. It is one of the hardest general-knowledge exams for AI. Leader: 1 / GPT 6 Astra / OpenAI / 54.8 / 39.0 / 2026-09-03. - [LiveBench](https://benchmarks.site/bots/internet/livebench.md) — A set of regularly refreshed tests (agent tasks, reasoning, coding, research, design, etc.) designed so AI models cannot have seen the questions beforehand. Leader: 1 / claude-fable-5.2 / 84.9 / 82.9 / 88.8 / 79.2. - [SimpleBench](https://benchmarks.site/bots/internet/simplebench.md) — Trick questions about everyday situations, space, time and people that ordinary adults usually get right but that can trip up AI. It tests common sense rather than book knowledge. Leader: 1 / Claude Fable 5.2 (unknown) / Anthropic / 94.9 / 2026-11-09. ## Math - [FrontierMath](https://benchmarks.site/bots/internet/frontiermath.md) — Hundreds of brand-new, unpublished math problems written by professional mathematicians, ranging from hard competition level up to problems that take experts days. Because the problems are secret, it is a strong check of real mathematical ability rather than memorization. Leader: 1 / Claude Fable 5.2 (max) / Anthropic / 95.1 / 100.0 / 2026-11-09. - [MathArena](https://benchmarks.site/bots/internet/matharena.md) — Tests AI models on fresh, hard math problems taken from new competitions and research papers, so the answers cannot have been memorised. Leader: 1 / Gemini 4 Carbon (high) / Google / 94.28% ± 3.64% / $5.08. - [PutnamBench](https://benchmarks.site/bots/internet/putnambench.md) — Tests whether AI systems can write machine-checkable formal proofs for problems from the Putnam, a famously hard undergraduate math competition. Leader: 1 / Aleph Prover (Logical Intelligence) / 672 / 2026-08-26 / Avg $74, Max $1468 per problem. ## Coding & software engineering - [SWE Atlas](https://benchmarks.site/bots/internet/swe-atlas.md) — Tests whether AI coding assistants can do everyday professional software work beyond fixing bugs: answering questions about a codebase, writing tests and refactoring code. Leader: 1 / Fable 5.2 / 80.55 / 4.80. - [SWE-Bench Pro](https://benchmarks.site/bots/internet/swe-bench-pro.md) — Tests whether AI agents can solve long, realistic software tasks in real open-source projects, harder than older bug-fixing tests. Leader: 1 / Opus 5 (Claude Code) xhigh / 98.00. - [SWE-rebench](https://benchmarks.site/bots/internet/swe-rebench.md) — Tests AI coding tools on fresh real-world bug-fix tasks collected after models were trained, so they cannot have seen the answers. Leader: 1 / Anthropic Fable 5 [high] / 64.5 / 78.4 / $4.40. - [Terminal-Bench](https://benchmarks.site/bots/internet/terminal-bench.md) — Tests whether AI agents can complete real tasks by typing commands in a computer terminal, like a developer or system administrator would. Leader: 1 / Opus 5.5 ( max ) / Claude Code / 65.8% ± 3.1% / $4.7k. ## Agents, tool use & computer use - [APEX-Agents](https://benchmarks.site/bots/internet/apex-agents.md) — Tests AI agents on long professional tasks like those of bankers, consultants and lawyers. Leader: 1 / Gemini 4 Carbon (High) / 84.2% ±4.3%. - [GDPval-AA](https://benchmarks.site/bots/internet/gdpval-aa.md) — Tests AI on real office-style professional tasks across many occupations, judged head-to-head. Leader: 1 / Claude Opus 5.5 (Max, Default Fallback) / 1866 / Sep 2026. - [MCPMark](https://benchmarks.site/bots/internet/mcpmark.md) — Tests AI agents on tasks that require using MCP tool servers such as Notion and GitHub. Leader: 1 / kimi-k3-max / 96.06. - [METR Time Horizons](https://benchmarks.site/bots/internet/metr-time-horizons.md) — Measures how long a task (in human working time) an AI can complete about half the time; shows how fast agents are improving. Leader: 1 / claude_mythos_preview_early_inspect / 1044.8 min (508.9-3304.3) / 2026-04-07. - [OSWorld (OSWorld-Verified)](https://benchmarks.site/bots/internet/osworld-osworld-verified.md) — Gives an AI control of a real computer (mouse, keyboard, screen) and asks it to do everyday jobs in apps like a browser, spreadsheet or image editor. It shows how close AI is to operating a PC the way a person does. Leader: 1 / Intelligence-Indeed Agent / Intelligence Indeed / Agentic framework / 100 / 90.2 / 2026-07-25. - [OSWorld 2.0](https://benchmarks.site/bots/internet/osworld-2-0.md) — A much harder follow-up to OSWorld: the AI must carry out long, multi-step office and professional workflows on a real computer, jobs that take a person about an hour and a half. It shows whether AI can handle long, realistic computer work rather than quick clicks. Leader: 1 / Claude Opus 5 / Anthropic / max / batch tool / v2.1 / 44.33 / 77.67 / yes. - [Remote Labor Index (RLI)](https://benchmarks.site/bots/internet/remote-labor-index-rli.md) — Tests whether AI can complete real paid freelance projects well enough that a client would accept the work. Leader: 1 / Fable 5.2 (NEW) / 41.25. - [ScreenSpot-Pro](https://benchmarks.site/bots/internet/screenspot-pro.md) — Shows an AI a screenshot of professional software (like CAD or video editors) and asks it to point to the right button or text. It tests whether an AI can "see" where to click, a basic step for computer-using agents. Leader: 1 / GroundCascade (Indeed-UI-32B + KV-Ground-8B) / 85.14 / 92.12 / 73.84. - [Toolathlon (Tool Decathlon)](https://benchmarks.site/bots/internet/toolathlon-tool-decathlon.md) — Tests AI agents on long, multi-step tasks across many everyday software tools. Leader: 1 / GLM 5.3 Flash (max) / 78.4 ± 1.9. - [Vending-Bench 2](https://benchmarks.site/bots/internet/vending-bench-2.md) — An AI runs a simulated vending-machine business for a long time; tests staying sensible over many days. Leader: 1 / Gemini 4 Carbon / $16,484.21 ± $1,096. ## Long context - [Context Arena (MRCR)](https://benchmarks.site/bots/internet/context-arena-mrcr.md) — Hides several near-identical items in a very long conversation and asks the AI to retrieve a specific one, checking whether it can find the right detail in a huge pile of text. Leader: 1 / openai/gpt-5.6-sol [max] / 92.4 / 85.5. ## Instruction following - [MultiChallenge](https://benchmarks.site/bots/internet/multichallenge.md) — Tests whether an AI can hold a realistic multi-turn conversation: remembering what you said earlier, following your instructions across turns, and reasoning over the whole chat. Leader: 1 / gpt 6.1 Sol / 81.95 / 4.16. ## Hallucination & factuality - [AA-Omniscience](https://benchmarks.site/bots/internet/aa-omniscience.md) — Tests how much factual knowledge an AI really has across 42 topics, and punishes it for confidently making things up instead of saying 'I don't know'. A score of 0 means as many wrong as right answers. Leader: 1 / Claude Fable 5.2 (Max, Default Fallback) / 67 / 79. - [Vectara Hallucination Leaderboard](https://benchmarks.site/bots/internet/vectara-hallucination-leaderboard.md) — Asks AI models to summarize short documents and uses an automated checker to count how often the summary adds things that are not in the original text. It shows how likely a model is to make things up when it should just be faithful to a source. Leader: 1 / antgroup/finix_s1_32b / 1.8 % / 98.2 % / 99.5 % / 172.4. ## Human-preference arenas - [Arena (formerly LMArena / Chatbot Arena)](https://benchmarks.site/bots/internet/arena-formerly-lmarena-chatbot-arena.md) — Real people type a question, get two anonymous answers from different AI chatbots, and vote for the better one. Millions of these votes are turned into a ranking of which chatbots people prefer. Leader: 1 / Gemini 4 Carbon (High) / Google / 1526 / ±10 / 4,273 / Proprietary. - [Design Arena](https://benchmarks.site/bots/internet/design-arena.md) — People ask several AI models to build the same thing, such as a website, game or slide deck, then vote on which result looks and works better. The votes become a ranking of which models are best at making designs and apps. Leader: 1 / Claude Opus 5.5 (high) / Anthropic / 1445 / 69.1% / 2,670. - [Artificial Analysis Image & Video Arenas](https://benchmarks.site/bots/internet/artificial-analysis-image-video-arenas.md) — People are shown two AI-made pictures (or videos) for the same request without knowing which model made them, and pick the better one. The votes become a ranking of image and video generators. ## Multimodal & vision - [MMMU / MMMU-Pro](https://benchmarks.site/bots/internet/mmmu-mmmu-pro.md) — College exam-style questions that come with pictures, such as charts, diagrams, maps and sheet music, across 30 subjects. It checks whether an AI can understand images and reason about them like a university student. Leader: 1 / Chance Vision 1.5 / proprietary / 2026-07-01 / 86.9 / -. ## OCR & document understanding - [OmniDocBench](https://benchmarks.site/bots/internet/omnidocbench.md) — Tests how well AI can read a PDF page and turn it into clean text, including tables and math formulas. It matters for turning scanned or complex documents into usable data. Leader: 1 / OXR-1.0 / Specialized VLMs / 1.3B / 97.03 / 0.0244 / 96.93 / 96.61. ## Speech recognition (ASR) - [Open ASR Leaderboard](https://benchmarks.site/bots/internet/open-asr-leaderboard.md) — Tests how accurately speech-to-text systems write down what people say in English recordings (meetings, earnings calls, audiobooks and more). Lower error means fewer wrong words. Leader: 1 / zoom/scribe_v2.1_pro / 4.87 / . - [Artificial Analysis Speech to Text](https://benchmarks.site/bots/internet/artificial-analysis-speech-to-text.md) — Compares services that turn spoken audio into written text: how many words they get wrong, how fast they are and what they cost. ## Text-to-speech - [TTS Arena V2](https://benchmarks.site/bots/internet/tts-arena-v2.md) — People listen to two computer-generated voices reading the same text and pick the one that sounds better. The ranking shows which text-to-speech systems people actually prefer. Leader: 1 / Aurora / 1687 / 73.4 / 312. - [Artificial Analysis Speech Arena](https://benchmarks.site/bots/internet/artificial-analysis-speech-arena.md) — Listeners hear the same sentence read by two AI voices without knowing which is which and pick the more natural one. The votes become a ranking of text-to-speech models. ## Safety & security - [CyberGym](https://benchmarks.site/bots/internet/cybergym.md) — Gives AI agents a description of a real security bug in open-source software plus the code, and asks them to write an input that actually triggers the bug. It shows how good AI has become at the hands-on work of finding and reproducing vulnerabilities, which helps defenders and could help attackers. Leader: 1 / Creation Model / Claude Code / 90.6% / 2026-10-21 / Creation Lab (Creation Lab, Ant Group). - [F5 Labs CASI & ARS Leaderboard](https://benchmarks.site/bots/internet/f5-labs-casi-ars-leaderboard.md) — Tests how hard it is to trick AI models into harmful behavior using attack prompts. A higher score means the model is harder to break. Leader: 1 / Anthropic / Claude Sonnet 5 / 98.00 / 23.20%. - [Scale Fortress](https://benchmarks.site/bots/internet/scale-fortress.md) — Tests whether AI models can be pushed into giving dangerous help (e.g. weapons or crime) by adversarial prompts. Lower is safer. Leader: 1 / claude-opus-5-5-max / 5.78±1.26. ## Multilingual - [EuroEval](https://benchmarks.site/bots/internet/euroeval.md) — Tests how well AI language models understand and write in more than 30 European languages, across many kinds of tasks. It matters because most AI tests are English-only. Leader: 1 / gpt-6-astra (zero-shot, val) / 1.17 ± 0.05 / No. ## Polish language - [CPTU-Bench](https://benchmarks.site/bots/internet/cptu-bench.md) — Checks how well AI models really understand Polish: reading between the lines (sarcasm, hidden meaning, idioms) and answering tricky questions with logic traps, ambiguity or absurd premises. It shows whether a model grasps Polish beyond the literal words. Leader: 1 / meta-models/Muse-Glimmer-30B reasoning (API, FP8) / 30B / 4.42 / 4.52 / 4.39 / 4.23 / 4.52. - [PL-MTEB (Polish MTEB)](https://benchmarks.site/bots/internet/pl-mteb-polish-mteb.md) — Tests how well "embedding" models (the part of AI search that turns text into numbers so similar meanings can be found) work on Polish text, on tasks like finding similar sentences, sorting reviews and grouping documents. Leader: 1 / codefuse-ai/F2LLM-v2-14B / 75.13 / 71.96 / 13.99. - [Polish Linguistic and Cultural Competency (PLCC)](https://benchmarks.site/bots/internet/polish-linguistic-and-cultural-competency-plcc.md) — 600 hand-written questions about Polish history, geography, culture and traditions, art and entertainment, grammar and vocabulary. It shows how well an AI model knows Poland and the Polish language, not just how fluent it sounds. Leader: 1 / Gemini-4-Carbon / Google / 98.33 / 97 / 100 / 100 / 97 / 98 / 98. ## Speed, price & providers - [OpenRouter Rankings](https://benchmarks.site/bots/internet/openrouter-rankings.md) — Shows which AI models people actually use most on OpenRouter, a service that lets developers switch between many AI models. It reflects real-world popularity, not test scores. Leader: 1 / deepseek/deepseek-v4.1-flash-20261017 / 61170.1 / 978.5. - [Artificial Analysis API Provider Benchmarks](https://benchmarks.site/bots/internet/artificial-analysis-api-provider-benchmarks.md) — Compares the companies that host AI models on speed, price and delay, so you can pick the fastest or cheapest provider for the same model. ## Aggregated leaderboards - [Epoch AI Benchmarking Hub](https://benchmarks.site/bots/internet/epoch-ai-benchmarking-hub.md) — Epoch AI runs AI models itself on a set of hard tests (maths, science, coding) and publishes the results in one place, so you can compare models on tests the labs did not grade themselves. Leader: 1 / claude-fable-5-2_max / Anthropic / 95.1 / 2026-11-09. - [Epoch Capabilities Index (ECI)](https://benchmarks.site/bots/internet/epoch-capabilities-index-eci.md) — One combined score for how capable an AI model is overall, built by stitching together results from many different tests so models from different years can be compared on one scale. Leader: 1 / Claude Fable 5.2 / Anthropic / 173.44 / API access / 2026-11-09. - [Kagi LLM Benchmarking Project](https://benchmarks.site/bots/internet/kagi-llm-benchmarking-project.md) — Kagi, a search company, regularly quizzes AI models on a private set of tricky reasoning, coding and language questions that change over time, to see which ones actually think well. Leader: 1 / claude-5-fable-thinking / 91.4 / 10.4 / kagi (ult). - [LLM Stats](https://benchmarks.site/bots/internet/llm-stats.md) — A website that gathers public test results, prices and speeds for hundreds of AI models and ranks them with a single combined score. Leader: 1 / Claude Fable 5.2 / Anthropic / 60.8 / 60.0 / 49.8 / 46.3. - [Scale SEAL Leaderboards](https://benchmarks.site/bots/internet/scale-seal-leaderboards.md) — Scale AI tests AI models on private, expert-written questions that models cannot have memorised, across areas like coding, reasoning and safety. One representative board (Humanity's Last Exam, Diamond) is shown. Leader: 1 / claude-fable-5-2 / anthropic / 72.4 / 2.8. - [Vals AI](https://benchmarks.site/bots/internet/vals-ai.md) — Vals AI tests AI models on realistic professional work in finance, law, tax and coding and combines the results into one index, aiming to show how useful models are for real jobs. Leader: 1 / Claude Fable 5.2 / 71.31 / $17.94 / 43m42s. - [Artificial Analysis Intelligence Index](https://benchmarks.site/bots/internet/artificial-analysis-intelligence-index.md) — Artificial Analysis combines results from a bundle of tough tests into one 'intelligence' score per AI model, alongside price and speed, to help people choose a model.