# Methodology & caveats

Every result on this site is "model (or engine) X on task Y", with the date, platform, sample size and conditions stated next to it. The pages describe the measurements only, not how any product uses them.

## Speech, vision and text tests

Applies to: Polish ASR, diarization, omni (video/audio) models, OCR of handwritten notes, boards and textbooks, language-model classification / summaries / chat, Polish embeddings, and platform availability. Measured September–November 2026.

### Metrics

- **WER** (word error rate): the share of erroneous words (substitutions + deletions + insertions) relative to the source text, after normalizing numbers and formulas read aloud (e.g. "1410" = „tysiąc czterysta dziesiąty”, "one thousand four hundred tenth"). Lower = better. Values above 100% are possible (loops, insertions).
- **CER** (character error rate): the same at the character level; for photos, computed on canonical text (flattened LaTeX, no formatting and no spaces).
- **Facts**: a list of regular expressions for things that must survive (dates, numbers, names, formulas, exercises). E.g. "105/109" = 105 of 109 facts found. For summaries: items of a knowledge list, scored by a judge model.
- **DER** (diarization error rate): the share of time with a wrongly assigned speaker, on 10 ms frames.
- **Accuracy** (embeddings, classification): the share of correct results on the test; for embeddings, additionally the "margin" = the lowest score of a relevant note minus the highest score of an irrelevant one.
- **Latency**: time from sending to the first token / to the end of the response, measured from Poland; depends on the platform's load at that moment.

### Test data

- ASR recordings: mostly **speech synthesis** (macOS `say`, Polish and foreign voices), sometimes deliberately degraded (reverb, noise, distance, heavy compression, classroom murmur). Levels: `clean`, `classroom` (phone on a desk in the front rows), `back` (phone at the back of a classroom, strong reverb). **No measurements on real classroom recordings.**
- OCR photos: **generated from handwriting fonts** (notebooks, boards, worksheets, textbook pages), with simulated rotation, shadow, noise and compression. Real notebooks will be harder.
- Summaries: four synthetic high-school lessons (mathematics, history, biology, Polish) with noise in the style of a speech-recognition transcript.
- No real recordings or private content of any person.

### Platforms

Most measurements via **DeepInfra** (OpenAI-compatible API, open models and some partner models); additionally **Soniox** (speech recognition, asynchronous and live) and **OpenAI** (transcription models, live mode). Google models (Gemini) on DeepInfra are forwarded to Google. The platform is given with every row.

### Dates

2026-09-22 (ASR, OCR, LLM, embeddings), 2026-09-23 (video/audio), 2026-09-29/30 (ASR and OCR on degraded data, languages, diarization, summaries), 2026-10-04…08 (photos, textbooks, illustrations), 2026-11-05…06 (live speech, provider ASR, chat). No measurements of speech synthesis (TTS) quality or voice-conversation models.

### Caveats

1. **Small samples.** Most rows are one to a dozen or so trials, one or two runs. Differences of 1–3 percentage points are within noise. The sample is given with each result.
2. **Synthetic data** inflates results compared with a real classroom (several voices, people speaking at the same time), although the `back` level is deliberately hard.
3. **Non-deterministic models** (especially Whisper) give different results for the same file; where this was measured, two runs are given.
4. **Platforms change**: queues, limits and model redirects occurred during the measurements. Results reflect the state on the given date.
5. **Judge model**: summaries were scored by a language model; it was sometimes too strict and raised false alarms.
6. "Best in test" applies to the specific task and set, not to the model's overall quality.
7. Own measurements, not independently audited.
8. Processing pipelines, model combinations and costs are deliberately out of scope here.

## Exam-sheet OCR, reasoning and text tests

Applies to: OCR of exam-sheet photos, reasoning effort, language models (short answers, grading text, time and cost), and platform errors / billing / prices. Measured September–October 2026.

### Metrics

- **Reading accuracy**: readings matching the reference out of attempts, e.g. "18/18". The reference was transcribed by hand from an enlarged photo. For sentences: word-for-word match, or the share of words in common (≥ 90%).
- **Cost**: the amount reported by the provider in the response (`estimated_cost`, in USD), converted to grosze (gr), not calculated from the token price list. This matters for models that bill reasoning tokens outside `completion_tokens`. "gr" is the grosz: 1 gr = 0.01 PLN.
- **Time**: from sending the request to receiving the full response (no streaming), from Poland. Depends on the platform's load at that moment.
- **Reasoning**: a model "reasons" when the response reports a non-zero number of reasoning tokens.
- Numbers that were not measured directly but reconstructed (from the total time of several calls, from change history) are marked **reconstruction** or **estimate**.

### Test data

- Photos: a few photos of Polish school exam-sheet pages (Polish language 2026, 1500×2000 px): one page with handwritten answers (two circled letters and one sentence of an open answer), four pages with no handwriting; in addition, a rendered blank mathematics exam sheet (2 pages) and sets of photos for identifying the exam sheet. **No measurements on many real handwritten papers.**
- Texts: short test queries, steps of a conversation with tools on a test account, answers to exam tasks.
- No personal data; handwritten content is not quoted apart from the single word under study.

### Platforms

Mostly **DeepInfra** (OpenAI-compatible API; Google Gemini and Anthropic Claude models are forwarded to Google and Anthropic). Some via **OpenAI** (gpt-5, gpt-5.6). Google **Document AI** was used briefly, with no accuracy measurements. The platform is given on every row.

### Dates

2026-09-03 (reasoning, model costs, Qwen3-VL variants), 2026-09-04 (blank page, gemma-4-E4B-it), 2026-09-17 (cost of a page reading with Gemini 3.1 Pro), approx. 2026-09-22/23 (exam-sheet identification, handwritten word 18/18), 2026-10-13 (reading handwriting and blank pages, DeepSeek-V4-Flash with and without a date, error 402), 2026-10-16 (DeepInfra model list and prices). No quality measurements for Claude Opus 5.5, Kimi, GLM, gpt-oss, or for the accuracy of answer grading.

### Caveats

1. **Very small samples.** Most rows are one photo and a few to a few dozen repetitions. A result applies to the specific photo, not to the model's general quality.
2. **No test of grading accuracy.** Nobody compared model grades with an examiner's grades on many papers.
3. **Some numbers are reconstructed** from total times, saved readings and change history; this is marked on the row.
4. **Models are non-deterministic**: the same photo produced different readings; results also depend on the wording of the prompt.
5. **Platforms change**: overloads, redirects between model variants and lack of account balance occurred during the measurements. Results reflect the state on the stated date.
6. Own measurements, not independently audited.
7. Processing pipelines, prompts and configuration are deliberately out of scope here.

## macOS malware-scanning tests

Applies to: the malware-detection pages, static-heuristic false alarms, ML classifiers on TLSH fingerprints, and scanner performance. Measured November 2026 on a macOS malware scanner under development. Only aggregates are published: counts, rates, per-category numbers, names of public test files and malware families, public SHA-256 hashes, and detection engine / signature names.

### Verdict levels and thresholds

Each file gets one of the verdicts `CLEAN`, `INFO`, `SUSPICIOUS`, `PUA` or `MALICIOUS`, based on a score summed from individual detections (heuristics, YARA rules, ClamAV / XProtect signatures, hash databases, fuzzy TLSH matching). Results are reported at three cumulative thresholds:

| Threshold | Counts as a hit | Meaning |
|---|---|---|
| INFO+ | `INFO`, `SUSPICIOUS`, `PUA`, `MALICIOUS` | Anything reported at all, including low-weight informational findings that are not shown as an alarm. |
| SUSPICIOUS+ | `SUSPICIOUS`, `PUA`, `MALICIOUS` | What a user would see as an alarm. This is the threshold used for the false-alarm target. |
| MALICIOUS | `MALICIOUS` | High-confidence detection only. |

In the held-out benchmark the counts are consistent with this: 318 `MALICIOUS` + 141 `SUSPICIOUS` + 20 `PUA` = 479 at SUSPICIOUS+, and 479 + 40 `INFO` = 519 at INFO+ (out of 528 test samples).

### Test data

- **Malicious test samples:** 528 files in seven groups — `yara/scripts_web` 312, `yara/gen` 107, `yara/macos` 60, `yara/native` 16, `scripts` 16, `documents` 15, `eicar` 2 — plus 14 clean test samples and 48 neutral archive files (no expected verdict) in the static-heuristics fixtures.
- **Clean files from a real Mac:** installed applications, Homebrew packages, Python libraries, application support data, installers, documents and dependencies. Only category names and counts are published.
- **Held-out clean set:** clean files are split into a set used while tuning heuristics (2309 files; 2077 in the 2026-11-09 run) and a held-out set never used for tuning (2973 files; 2812 in the 2026-11-09 run). The held-out false-alarm rate is the honest number; the tuning-set rate is shown only for comparison.
- **Safe benchmark:** only public, harmless test files (eicar.org, AMTSO Security Features Check) and public data (SHA-256 hashes from Objective-See reports, family names).

### Caveats

- Sample sizes are small for some categories (e.g. `project_deps` 4 files, `user_documents` 22, `installers_archives` 30, clean test samples 14); a single file changes the rate by several percent.
- Zero false alarms at SUSPICIOUS+ on 2973 held-out files does not mean a zero rate: by the rule of three the 95% upper bound is about 3/2973 = 0.10%.
- 495 of the 528 malicious test samples are in the `yara/*` groups (test fixtures for the YARA rule set), so the detection rate is not a measure against unknown, fresh malware. The Objective-See hash test (8/33) and the family coverage test (57/83) show the gaps on real-world Mac malware more directly.
- The hash and family tests check database / rule coverage, not scanning of live samples.
- All measurements were taken on a single Mac; timings are wall-clock and were not taken under controlled load. Manual performance measurements have no raw result files.
- Not measured yet: local LLMs (Foundation-Sec vs others), databases (SQLite / DuckDB / LMDB / RocksDB), and other antivirus engines (ClamAV `clamscan`) on the safe benchmark.
