# benchmarks.site > Independent AI model and platform benchmarks on real tasks (OCR of handwriting and documents, Polish speech recognition, diarization, language models, reasoning effort, embeddings, platform behavior, macOS malware detection), plus a curated directory of public AI benchmarks. Every result states date, platform and sample size. Read the methodology and caveats before citing. Cite as: benchmarks.site, with the page URL and measurement date. ## Our benchmarks - [Methodology & caveats (read before citing)](https://benchmarks.site/bots/methodology.md): How the results were measured, what the test data is, and what the numbers do and do not show. Read before citing. - [ASR: Polish speech recognition](https://benchmarks.site/bots/asr-polish.md): On clean synthetic speech, Qwen3-ASR-1.7B and Gemini 3.1 flash-lite kept Polish–English mixing verbatim while Whisper-large-v3 translated it; on degraded recordings the external providers Soniox stt-async-v6 and OpenAI gpt-transcribe had far lower WER (8.9% and 14.0% at the back-of-classroom level vs 34.4–63.8% for the DeepInfra models), and Soniox stt-rt-v6 had the lowest WER in live streaming. All test audio is synthetic. - [Diarization: who is speaking](https://benchmarks.site/bots/diarization.md): On four short synthetic lessons, nvidia/Nemotron-3-Diarization (DeepInfra) averaged 9.4% DER and detected 31 of 36 overlapping-speech events, while Soniox stt-async-v6 and OpenAI gpt-4o-transcribe-diarize detected none; note that a trivial "one speaker" answer already scores 13–18% DER on this set. - [Omni models: video and audio input](https://benchmarks.site/bots/omni-video-audio.md): On DeepInfra, `video_url` input delivers only image frames to the model (audio was dropped for 6 of 6 models), and models see a fixed number of frames (32 or 16 where measured) regardless of video length; audio has to be sent as a separate `input_audio` part (wav/mp3 only, max 600 s per part). - [OCR: handwritten notes, boards and textbook pages](https://benchmarks.site/bots/ocr-handwriting-boards-textbooks.md): On synthetic Polish notes, handwriting and textbook pages, Kimi-K2.6, Qwen3.5-397B-A17B and DeepSeek-V4.1-Flash were among the most accurate single models, and on print all tested models stayed below 1.5% CER. Small numbers are the weak spot - on one test-schedule page every model "corrected" 2026 to 2025 - and on tiny or distant writing models reconstruct content instead of reading it. - [OCR: exam-sheet photos with handwritten answers](https://benchmarks.site/bots/ocr-exam-sheets.md): Gemini 3.1 Pro read circled answers 80/80 and did not invent answers on blank pages (12/12), but misread a handwritten word toward the printed word on the same page depending on the prompt; Qwen3-VL-235B invented answers on a blank sheet during full-page reading yet answered a single yes/no "is there handwriting" question correctly. Samples are very small. - [Language models: classification, summaries, chat with tools](https://benchmarks.site/bots/llm-classification-summaries-chat.md): DeepSeek-V4.1-Flash was the most consistent choice across lesson classification, faithful lesson summaries (99.6–99.7% of knowledge items kept, no truncated notes) and streaming tool chat. Abbreviations like EDB and WOS confuse cheap classifiers unless explained in the prompt, and reasoning toggles differ between models. - [Language models: short answers, grading text, time and cost](https://benchmarks.site/bots/llm-short-answers-grading-cost.md): On DeepInfra, the undated name deepseek-ai/DeepSeek-V4-Flash hit an overloaded pool (7/12 requests rejected with HTTP 429), while the dated DeepSeek-V4-Flash-0731 answered every time and faster; automatic prompt caching cut the cost of a long step approx. 3.6×. No grading-accuracy test was run. - [Reasoning level (reasoning_effort): what models actually do](https://benchmarks.site/bots/reasoning-effort.md): On `minimal`, only DeepSeek-V4-Pro actually produced reasoning tokens; gpt-5 accepts `minimal` but does not reason and gpt-5.6 rejects it. Reasoning is expensive (DeepSeek-V4-Flash 7.3× more with reasoning) and can consume the whole output limit, yielding empty responses (gpt-5.6 at a 2000-token limit). - [Embeddings: retrieval accuracy in Polish](https://benchmarks.site/bots/embeddings-polish.md): On a small Polish note set, the Qwen3-Embedding models (0.6B, 4B, 8B) ranked relevant notes above lookalike distractors, with Qwen3-Embedding-0.6B giving the best margin and 10/10 queries with an instruction prefix; bge-m3, multilingual-e5-large and embeddinggemma-300m let unrelated notes outrank a relevant one. - [Platforms: availability, latency and limits](https://benchmarks.site/bots/platforms-availability-latency-limits.md): Always check the `model` field in responses (DeepInfra silently redirects retired models to replacements), expect load-dependent latency and queues on DeepInfra, and note Soniox's default limit of 10 parallel streams per organization; OpenAI live transcription provides no end-of-utterance marker or word timestamps. - [Platforms: errors, reasoning billing and prices](https://benchmarks.site/bots/platforms-errors-billing-prices.md): On DeepInfra, check the `model` field and reasoning-token counts (a "Thinking" variant was served by Instruct), prefer dated model names (undated DeepSeek-V4-Flash was often overloaded), and do not trust price-list cost estimates for Gemini (approx. 13× too low); 402/404/429 codes and error text distinguish account limits, missing models and pool overload. - [Malware detection: AMTSO, EICAR containers, Objective-See hashes, Mac malware families](https://benchmarks.site/bots/malware-detection-safe.md): All 11 official AMTSO / eicar.org test files and EICAR in all 57 container formats are detected with no alarm on the 57 clean twins; coverage of real Mac malware is much lower — 8/33 Objective-See hashes are in the database and 57/83 families (2019–2025) have a family-named rule. - [Malware detection: test samples vs held-out clean files](https://benchmarks.site/bots/malware-detection-held-out.md): The full engine detects 519/528 test samples at INFO+, 479/528 at SUSPICIOUS+ and 318/528 at MALICIOUS, with 0 SUSPICIOUS+ false alarms on clean files never used for tuning (0/2973, later 0/2812); a fix of two context heuristics cut held-out INFO+ findings from 119/2973 (4.00%) to 10/2973 (0.34%). - [Malware scanning: static-heuristic false alarms before and after fixes](https://benchmarks.site/bots/malware-static-false-positives.md): Fixes to the static heuristics removed all 6 SUSPICIOUS false alarms on 2945 clean Mac files (0.20% → 0/2944) while raising heuristic-only detection on the 528 test samples from 107 to 135 at SUSPICIOUS+; an intermediate change that briefly added 2 new false alarms was reverted. - [ML classifiers on TLSH fingerprints of Mach-O files](https://benchmarks.site/bots/ml-tlsh-macho-classifiers.md): Gradient-boosted trees are the best model on TLSH features — GBDT 600×d6 reaches out-of-fold AUC 0.966 and 39.3% recall at a 0.1% false-positive rate, ahead of MLPs (18–24%) and logistic regression (9.5%); nearest-neighbour TLSH distance rules alone are far weaker (AUC 0.582). - [Scanner performance: database loading, re-scans, TLSH in SQL, Aho-Corasick](https://benchmarks.site/bots/scanner-performance.md): Database load time dropped from 60–77 s to ~11 s, a re-scan of Numbers.app (33819 files) from 131.7 s to 24.7 s, and moving TLSH candidate selection into SQL gave identical results (718/718) ~1.6× faster; `ahocorasick_rs` built lazily beats loading a pickled `pyahocorasick` automaton, and threaded sharding gives no speedup because of the GIL. ## Benchmarks from the internet - [Public leaderboards with results](https://benchmarks.site/bots/internet.md): 52 public benchmarks explained in plain words, 47 with results copied here (source stated). One Markdown file per benchmark under https://benchmarks.site/bots/internet/. - [Best open-weights models](https://benchmarks.site/bots/open-models.md): open models ranked across public leaderboards (approximate). - [Rankings](https://benchmarks.site/bots/rankings.md): which models finish first most often across public leaderboards (approximate). - [Glossary](https://benchmarks.site/bots/glossary.md): plain-language meanings of WER, CER, Elo, tokens, embeddings and more. - [Public leaderboards as JSON](https://benchmarks.site/bots/internet.json) ## Contact - Sponsorship, benchmark requests, corrections: filip@biskupski.site (https://benchmarks.site/contact/) ## Data - [All result tables as JSON](https://benchmarks.site/bots/tables.json): every table with benchmark, section, columns and rows. - [Everything in one file](https://benchmarks.site/llms-full.txt): all benchmark pages and the directory as one Markdown file.