# ASR: Polish speech recognition

> On clean synthetic speech, Qwen3-ASR-1.7B and Gemini 3.1 flash-lite kept Polish–English mixing verbatim while Whisper-large-v3 translated it; on degraded recordings the external providers Soniox stt-async-v6 and OpenAI gpt-transcribe had far lower WER (8.9% and 14.0% at the back-of-classroom level vs 34.4–63.8% for the DeepInfra models), and Soniox stt-rt-v6 had the lowest WER in live streaming. All test audio is synthetic.

Area: Speech. Page: https://benchmarks.site/asr-polish/. Methodology and caveats: https://benchmarks.site/methodology/#speech-vision-text

# Speech recognition (ASR) in Polish

All rows are results of **single models**, without text correction by a second model and without merging transcripts. Metrics: WER (lower = better), facts (how many of 109 fact expressions were found in the transcript), handling of Polish–English mixing. Metric definitions and caveats: [overview](/methodology/#speech-vision-text).

## 1. Test of 2026-09-22: clean speech synthesis, Polish + English in one recording

- Platform: DeepInfra (Gemini: via DeepInfra to Google).
- Data: macOS synthetic speech (Zosia pl_PL, Samantha en_US), 16 kHz mono. Clips: `pl` 48.5 s math lecture with formulas and numbers; `en` 29.4 s biology in English; `mix` 20.1 s Polish-English-Polish; `mixEN1` 20.1 s English first; `noisy` = the `mix` clip + pink noise and 100 Hz hum, approx. 10 dB SNR (mp3 32 kb/s and Opus); `long` 45.85 min (repeated clips with silence).
- 178 calls in total. WER is "approximate" (light normalization of numbers and formulas). In the mixed clip, a translated or omitted English sentence counts as approx. 45% WER.
- A test on **clean speech synthesis** says nothing about a real classroom (see section 2).

| model | platform | WER PL clean | WER EN | mix clean, no language hint | mix noisy | runs with English preserved (verbatim) | latency, 20–48 s clip | 45-min file in one request |
|---|---|---|---|---|---|---|---|---|
| Qwen/Qwen3-ASR-1.7B | DeepInfra | 2.5% | 0% | 6.4% (verbatim) | 12.8% | 17/17 | 1.8–9 s | 15–27 s, 42/42 English sentences preserved |
| Qwen/Qwen3-ASR-0.6B | DeepInfra | 18.0% | 0% | 8.5% | 21–23% | 12/12 | 1.6–5.5 s | not tested |
| openai/whisper-large-v3-turbo | DeepInfra | 4.9% (drops words at the 30 s window boundary) | 0% | 2.1% when correct, 45–55% when hallucinating | 2–4% | 10/22 | 1.5–3.9 s (up to 10 s) | 13.5 s, only 21/42 English sentences preserved; 88 MB WAV file accepted |
| openai/whisper-large-v3 | DeepInfra | 6.6% | 0% | 44.7% (English sentence translated into Polish every time) | 46.8% (English omitted) | 0/12 | 5.5–23 s | not tested |
| mistralai/Voxtral-Small-24B-2507 | DeepInfra | 77% without a language hint (translates into English); 4.9% with Polish specified, but wrong month („maja” ("of May")) | 0% | 44.7% (English omitted) | 2.1% or 44.7% (inconsistent) | 0/10 | 3–8.8 s | not tested |
| mistralai/Voxtral-Mini-3B-2507 | DeepInfra | 64.8% without a language hint (output in Russian); 9% with a hint (output in LaTeX) | 1.2% | 76.6% (everything in English) | 94–96% | 0/10 | 1.5–3.3 s | not tested |
| nvidia/Nemotron-3.5-ASR-Streaming-Multilingual-0.6b | DeepInfra | 47.5% (40% with Polish specified) | 0% | 23.4% (writes in Cyrillic) | 38% | 6/12 (only with the language specified) | 1.6–5.6 s | not tested |
| google/gemini-3.1-flash-lite (chat model with audio input) | DeepInfra → Google | 0–1% | 0% | 0% (verbatim) | 0% | 3/3 | 2.5–3.4 s; 10 min in 10.5 s | 46 min without streaming: 300 s timeout exceeded |
| google/gemini-3.5-flash (chat model with audio input) | DeepInfra → Google | approx. 1% | n/a | n/a | 0% | 1/1 | 2.7–3.6 s | n/a |

Noisy mixed clip: Qwen3-ASR-1.7B has 12.8%, Gemini 3.1 flash-lite 2.1%; example of a Qwen error: „krótki fragment tekstu” ("a short piece of text") → „trubki Pradmentowskiego” (nonsense).

Observations (samples as above):
- Whisper-large-v3 picks one language per file and translates the rest: 12 of 12 mixed runs (English sentence translated into Polish); when English came first, it translated the Polish.
- Whisper-large-v3-turbo is non-deterministic even at temperature 0: in 11 of 22 runs it replaced the English sentence with a hallucination („Wszystkie prawa zastrzeżone” ("All rights reserved"), „Współpraca z Wrocławiem” ("Cooperation with Wrocław"), loops of „w tym roku” ("this year")). Both Whispers drop words at the 30 s window boundary („Delta wynosi dwadzieścia pięć” ("Delta equals twenty-five") → „5-24”). An additional text prompt for turbo emptied the output on the noisy clip.
- Qwen3-ASR gives identical output on repeats and returns word timestamps from forced alignment (English sentence started at 7.44 s vs the true 7.40 s), but has coarse segments (one per 20–27 s) and reports only the dominant language.
- Voxtral returns no timestamps or language; the webm and m4a formats end in an HTTP 500 error; the model is not available through the chat interface (404).
- Gemini: only mp3 and wav (ogg: error 422); no word timestamps; the 1024-token output limit truncated a 10-minute chunk.
- Formats: mp3, ogg/opus, wav, flac accepted by all 7 ASR models; webm/opus and m4a/aac by Whisper, Qwen and Nemotron.

## 2. Degraded recordings, 2026-09-29 (single models, no text correction)

- Platform: DeepInfra.
- Data: 8 topics (mathematics, physics, history, chemistry, biology, English PL+EN, Polish, civics (WOS)), approx. 9 minutes in total, speech synthesis, 109 facts (dates, numbers, names, formulas). Formulas and numbers read aloud. Three levels: `clean` (microphone at the mouth), `classroom` (phone on a desk in the front rows: reverb RT60 0.6 s, SNR 20 dB, AAC compression 64 kb/s), `back` (phone at the back of the classroom: reverb 0.85 s, SNR 14 dB, classroom murmur, AAC compression 48 kb/s).
- One run per cell unless noted otherwise. Depending on the test variant, Qwen's result at the `back` level fell within 58.5–64.1%, and Whisper v3's within 31.5–47.4%.

| model | platform | clean: WER (facts/109) | classroom: WER (facts/109) | back: WER (facts/109) | notes |
|---|---|---|---|---|---|
| Qwen/Qwen3-ASR-1.7B | DeepInfra | 10.9% (80) | 25.8% (61) | 63.8% (20) | deterministic; keeps English verbatim; in reverb loses about two thirds of the words |
| Qwen/Qwen3-ASR-0.6B | DeepInfra | 26.8% (60) | n/a | 88.1% (0) | unfit for Polish speech in reverb |
| openai/whisper-large-v3 | DeepInfra | 13.3% (89)\* | 15.8% (82)\* | 34.4% (51) in one run, 47.4% (50) in the second | \*one 30-s window lost; non-deterministic; translates English sentences into Polish; once every few runs turns a whole window into gibberish or drops it (topic "history" 55–71% WER even on clean) |
| openai/whisper-large-v3-turbo | DeepInfra | n/a | n/a | 41.9% (47) | one run |
| MiMo-V2.6-Pro and MiMo-V2.6-Flash (chat model, audio in the message) | DeepInfra | n/a | n/a | > 800% | loops („w tym samym roku…” ("in the same year…")) |
| Inkling, Inkling-Small, gemma-4-31B-it | DeepInfra | n/a | n/a | n/a | the interface rejects audio input (HTTP 400) |

Additionally: longer chunks for Whisper (60 s instead of ≤ 30 s) gave 45.8% WER at the `back` level (more dropped words). The DeepInfra model list as of 2026-09-29 contained no new ASR models beyond Whisper, Qwen3-ASR, Voxtral and Nemotron.

### Earlier measurement, 2026-09-23 (one deliberately degraded recording)

A recording imitating a lesson heard from a desk: reverb, pink noise, distance, 48 kb/s compression. WER against the source text, 1 recording, no text correction. The source does not state precisely whether the audio was preprocessed before models other than Qwen.

| model | platform | WER PL | WER PL+mix (average) | English preserved | notes |
|---|---|---|---|---|---|
| Qwen/Qwen3-ASR-1.7B, raw audio | DeepInfra | 19.0% | 26.0% | 71% | |
| openai/whisper-large-v3-turbo | DeepInfra | 23.3% | n/a | n/a | |
| openai/whisper-large-v3 | DeepInfra | 24.8% | n/a | n/a | |
| Voxtral Mini and Small | DeepInfra | ≥ 90% | n/a | n/a | unfit for Polish |
| Nemotron-Diarization (used as ASR) | DeepInfra | ≥ 90% | n/a | n/a | a speaker-recognition model, not a transcription model |

## 3. Language lessons: Polish and a foreign language in one sentence, 2026-09-29/30

- Platform: DeepInfra.
- Data: 4 lessons, 7.6 min in total: Spanish ×2 (pretérito indefinido; „en el restaurante”), English (present perfect), German (Perfekt). The Polish parts are read by synthetic Zosia, the foreign parts by synthetic native voices, so a sentence switches language in the middle. Averages over runs (Whisper run twice). Small sample: 4 lessons.
- "Preserved" = the foreign word written as spoken; "translated" = written in Polish; "garbled" = in the original language with errors; "lost" = missing, gibberish or phonetic spelling in Polish letters. Non-preserved fragments were scored by a judge model.

All languages together, single models:

| model | level | WER overall | WER of Polish fragments | foreign preserved | translated | garbled | lost |
|---|---|---|---|---|---|---|---|
| Qwen3-ASR-1.7B | clean | 15.4% | 16.0% | 86.1% | 0.5% | 5.8% | 7.7% |
| whisper-large-v3 | clean | 34.3% | 10.8% | 41.6% | 30.7% | 7.9% | 19.8% |
| Qwen3-ASR-1.7B | classroom | 35.6% | 31.7% | 62.1% | 10.9% | 10.9% | 16.1% |
| whisper-large-v3 | classroom | 33.4% | 29.9% | 68.4% | 19.3% | 8.4% | 4.0% |
| Qwen3-ASR-1.7B | back | 65.5% | 67.6% | 40.5% | 7.7% | 13.4% | 38.4% |
| whisper-large-v3 | back | 56.5% | 60.1% | 56.1% | 7.8% | 7.3% | 28.8% |

Whisper-large-v3 by foreign language (classroom / back):

| language | WER classroom | foreign preserved / translated, classroom | WER back | foreign preserved / translated, back |
|---|---|---|---|---|
| Spanish | 39.1% | 60.1% / 27.2% | 63.9% | 44.3% / 4.5% |
| English | 14.5% | 90.3% / 2.6% | 32.1% | 85.1% / 0.0% |
| German | 41.7% | 61.2% / 21.4% | 67.3% | 48.5% / 23.3% |

Observations: Qwen3-ASR keeps both languages, but in noise it writes foreign words phonetically in Polish („a djeł, lasy na paszare” instead of „ayer, la semana pasada”). Whisper picks one language in every 30 s window and translates or drops the other (a Spanish reading passage came out as „Wczoraj wyszłam do kina z moimi przyjaciółmi” ("Yesterday I went to the cinema with my friends"), in Polish). German at the back of the room was unreadable for Whisper (WER approx. 67%).

## 4. External providers on the same recordings, 2026-11-06

- Platforms: Soniox (`stt-async-v6`, file mode), OpenAI (transcription models).
- Data: the same 9 minutes, 8 topics, 109 facts, three levels (clean / classroom / back); plus 79 numbers and 54 proper names. Speech synthesis. Single model without text correction; the number of runs for these rows is not given in the source.
- WER and count columns: clean / classroom / back.

| model | platform | WER % | facts out of 109 | numbers out of 79 | names out of 54 | compute time, s per minute of recording | word timestamps |
|---|---|---|---|---|---|---|---|
| Soniox stt-async-v6 | Soniox | 2.4 / 3.4 / 8.9 | 103 / 97 / 87 | 77 / 77 / 77 | 52 / 50 / 42 | 8.6–14.8 | yes |
| gpt-transcribe | OpenAI | 1.2 / 4.7 / 14.0 | 107 / 104 / 92 | 78 / 77 / 75 | 54 / 53 / 43 | 2.8–4.3 | no |
| gpt-4o-transcribe | OpenAI | 16.7 / 16.8 / 30.7 | 79 / 76 / 65 | 67 / 67 / 67 | 27 / 29 / 19 | 3.5–5.6 | no |
| gpt-4o-mini-transcribe | OpenAI | 16.4 / 19.8 / 61.5 | 82 / 82 / 51 | 68 / 70 / 55 | 29 / 39 / 15 | 2.3–2.7 | no |
| gpt-4o-transcribe-diarize (classroom level only) | OpenAI | 11.2 | 88 | 77 | 44 | 29.9 | segments |

Notes: the `gpt-4o-*` models were measured with the language specified; without it, at the `classroom` level they did no better (18.6% and 20.3%). OpenAI has announced the shutdown of the `gpt-4o-*` models on 2027-02-26. Language lessons (mixed sentences) were not measured with these providers.

Processing time for a 45.6-min lesson (the classroom-level recording repeated 5 times, one measurement): Soniox in file mode, 78 s.

## 5. Live speech (streaming), 2026-11-05/06

- Platforms: Soniox (WebSocket), OpenAI (live mode), DeepInfra (Nemotron and Qwen3-ASR in 20 s chunks).
- Data: 3 lessons from fixtures (history / mathematics / biology), speech synthesis; the text contains fixture markers that are read aloud, which creates a floor of approx. 6–10% WER for all models, so this set barely shows differences between the good models. Nemotron and Qwen received separate 20 s chunks, the others a continuous stream (incomplete comparison).

| model | platform | WER clean (hist./math/biol.) | WER classroom (hist./math/biol.) |
|---|---|---|---|
| Nemotron 3.5 ASR Multilingual, language specified (20 s chunks) | DeepInfra | 26.5 / 29.5 / 31.4% | 43.7 / 39.9 / 46.3% |
| Qwen3-ASR (20 s chunks) | DeepInfra | 17.2 / 11.3 / 69.9%\* | 28.5 / 16.4 / 24.2% |
| Soniox stt-rt-v3 | Soniox | 9.3 / 6.4 / 6.9% | 10.7 / 6.4 / 8.2% |
| Soniox stt-rt-v6 | Soniox | 8.2 / 6.4 / 6.6% | 7.6 / 6.2 / 7.4% |
| OpenAI gpt-live-transcribe | OpenAI | 9.0 / 10.2 / 10.9% | 12.7 / 9.4 / 12.8% |
| OpenAI gpt-realtime-whisper | OpenAI | 9.3 / 10.5 / 10.4% | 11.5 / 10.2 / 9.8% |

\* Outlier run of Qwen; the other runs were 11–28%. Nemotron without the language specified: 37–45% WER and output in Cyrillic.

Latency and additional measures (link with no other traffic, at most 2–3 streams at once):

| model | first partial result | end of utterance → result (min / median / max) | WER at pauses | numbers / names from the lesson |
|---|---|---|---|---|
| Soniox stt-rt-v3, PCM | 961–1009 ms | 280 / 499 / 1709 ms | 4.0% | 168/202, 65/68 |
| Soniox stt-rt-v6, PCM | 1254–1316 ms | 254 / 461 / 691 ms | 3.1% | 167/202, 64/68 |
| OpenAI gpt-live-transcribe | 861–3296 ms under load; 1268–1337 ms at pauses | 185 / 484 / 1643 ms (last text) | 5.0% | 141/202, 68/68 |
| OpenAI gpt-realtime-whisper | 1503–7157 ms under load; 1673–1769 ms at pauses | 598 / 837 / 1310 ms (last text) | 6.6% | 138/202, 68/68 |

- Soniox provides an end-of-utterance marker and word timestamps; OpenAI in live mode did not provide them.
- Language hint (Soniox, average WER across lessons): `stt-rt-v3` with a list of languages 8.0%, `pl` only 8.5%, no hint 9.6%; `stt-rt-v6` with a list 7.1%, `pl` only 7.4%. Differences are approximate (some runs went over a loaded link).
- Foreign words pronounced the Polish way inside a Polish sentence (Soniox `stt-rt-v6`, 2026-11-10, 6 synthetic recordings) came out wrong, e.g. "altog" instead of "although". Adding a list of difficult words to the context changed the result from 3/6 to 6/6 correct recordings (very small sample).
- Nemotron on DeepInfra returns no partial results (despite "Streaming" in its name): a 3–6 s window takes 1.0–1.9 s of recognition alone (median 1.3 s). A 20 s chunk: 1.4–3.5 s (Nemotron) vs 50–77 s (Qwen3-ASR, under DeepInfra load on 2026-11-05); 10 chunks at once: 7–22 s vs 61–148 s.
