benchmarks.site

Speech

Diarization: who is speaking

Can a model tell who is speaking when, in a recording with several people?

Key finding

On four short synthetic lessons, nvidia/Nemotron-3-Diarization (DeepInfra) averaged 9.4% DER and detected 31 of 36 overlapping-speech events, while Soniox stt-async-v6 and OpenAI gpt-4o-transcribe-diarize detected none; note that a trivial "one speaker" answer already scores 13–18% DER on this set.

How to read the numbers

DER = share of time assigned to the wrong speaker. Lower is better.

Hover a column name for its meaning, or see the glossary.

How we tested & caveats · Markdown · CSV

Metric: DER (diarization error rate; lower = better), computed on 10 ms frames, including speech by several people at once, consistent with pyannote.metrics to within 0.1–0.2 points. Reference point: the trivial answer "all speech belongs to one speaker" gives a DER of 13–18% on this set (the teacher speaks 82–87% of the time), so a DER of 6–14% looks good mainly because one speaker dominates. Metric definitions and caveats: overview.

1. nvidia/Nemotron-3-Diarization (DeepInfra), 2026-09-30

  • The model returns only speaker turns (anonymous labels speaker_N, up to 8 speakers, consistent within a file, 10 ms resolution, no confidence and no text); turns of different speakers overlap when several people speak at once. The output is deterministic (the same on every call on the same file).
  • Data: 4 synthetic lessons, 7.1 min in total: history, biology, mathematics, English; 3–4 students in each, 4–5 places with two people speaking at once. In three lessons the students are a pitch-shifted (higher) synthetic Zosia, so the voices differ mainly in pitch (a hard case); the English lesson has clearly different voices. The classroom and back levels are as in ASR (section 2). Synthetic speech, no real children.
  • "Groups / speakers" = number of detected voice groups / number of true speakers. "Overlap: recall / precision" = frames in which several people speak at once. In parentheses at the back level: DER according to the source with a 0.25 s collar; for classroom the source gives a single value.
8 rows · 6 columnsCSV
historyclassroom6.8%3 / 494% / 94%5/5
biologyclassroom6.2%3 / 598% / 90%4/4
mathematicsclassroom11.5%3 / 582% / 88%3/4
Englishclassroom11.7%4 / 575% / 88%4/5
historyback8.3% (5.5%)3 / 484% / 87%4/5
biologyback6.5% (4.4%)3 / 598% / 88%4/4
mathematicsback13.5% (9.6%)2 / 585% / 92%3/4
Englishback11.1% (7.0%)4 / 591% / 87%4/5

Conclusions:

  • Speech by several people at once is a strong point: 75–98% of frames, 31 of 36 events detected.
  • Students whose voices differed only in pitch (pitch-shifted TTS) were sometimes merged into one group or attached to the teacher: the model distinguished 0–1 of 3–4 students, and short answers („Czterdzieści dziewięć.” ("Forty-nine.")) went to the teacher's group. With clearly different voices (English) it found 4 groups for 5 people. Results on real children may differ.
  • Time and scale: a 45-minute lesson (21.9 MB mp3) in one request: 16.7 s, 4 groups, 189 turns.
  • The model is a streaming model (with speaker memory); the file was not split.

2. Comparison with external providers, 2026-11-06

The same 4 lessons, classroom and back levels, the same DER measure. Results are for single models; WER is not given because it does not apply to diarization itself.

3 rows · 5 columnsCSV
nvidia/Nemotron-3-DiarizationDeepInfra9.4% (classroom 9.1%)31 of 36
Soniox stt-async-v6 (diarization in file mode)Soniox13.6% (classroom 12.7%)0 of 36at the level of the "one speaker" answer; does not capture speech by several people
OpenAI gpt-4o-transcribe-diarize (classroom only)OpenAI18.1%0 of 18provides labeled segments; processing time 29.9 s per minute of recording

Limitations: one run per model, small sample (4 short lessons), synthetic speech.