benchmarks.site

Speech

Omni models: video and audio input

Models that claim to accept video and sound: do they really hear the audio and see the frames?

Key finding

On DeepInfra, `video_url` input delivers only image frames to the model (audio was dropped for 6 of 6 models), and models see a fixed number of frames (32 or 16 where measured) regardless of video length; audio has to be sent as a separate `input_audio` part (wav/mp3 only, max 600 s per part).

How to read the numbers

Mostly yes/no checks and counts of what the model actually received.

Hover a column name for its meaning, or see the glossary.

How we tested & caveats · Markdown · CSV

Platform: DeepInfra. Measured on 2026-09-23 against the live API, supplemented with results from 2026-09-29. All numbers come from API responses (usage), not from documentation. Metric definitions and caveats: overview.

Catalog (2026-09-23)

From the DeepInfra model list: 232 models not marked as deprecated, including 17 with a video-input tag, 11 with an audio-input tag and 8 native ASR models. The tag is binding: a model without the tag responds with HTTP 405 ("does not accept video input"); e.g. google/gemini-3.1-flash-lite does not accept video (audio only). All Gemini models on DeepInfra are forwarded to Google (partner flag); google/gemma-4-31B-it-Ultra is also a partner model (despite open weights), while the plain gemma-4-31B-it and -turbo are hosted by DeepInfra.

1. Does the model hear the audio from a video file?

  • Test clip: 7.7 s, a slide showing x + 7 = 12 / Zadanie domowe: str. 42 ("Homework: p. 42"), a speech track with a "secret word" („bakłażan” ("eggplant")). Prompt: transcribe the board and quote what can be heard; if nothing can be heard, write BRAK DZWIEKU ("NO SOUND"). 1 clip, 1 trial per model.
6 rows · 3 columnsCSV
google/gemma-4-31B-it (video + audio)yesno audio
XiaomiMiMo/MiMo-V2.6-Flash (video + audio)yesno audio
XiaomiMiMo/MiMo-V2.6-Pro (video + audio)yesno audio
Qwen/Qwen3.5-35B-A3Byesno audio
inclusionAI/Ling-3.0-flash-VLyesno audio
zai-org/GLM-5.3-Flashyesno audio

Conclusion: on DeepInfra, video_url input delivers only image frames to the model; the audio is discarded (six models out of six). Evidence: the same MiMo-V2.6-Flash transcribed the same utterance word for word when it received it as a separate audio part, and with video and audio in one message it responded to both.

2. How many frames does the model see?

The number of input tokens for video was identical for videos of 30 s, 60 s, 300 s, 720 s and 2700 s, and for 1280×720 and 640×360 – a fixed number of frames regardless of length. No video length limit was found (a 2700 s video went through), nor any way to set the number of frames.

6 rows · 4 columnsCSV
google/gemma-4-31B-it242126632
inclusionAI/Ling-3.0-flash-VLapprox. 4400883approx. 32
Qwen/Qwen3.5-35B-A3Bapprox. 11720883approx. 32
XiaomiMiMo/MiMo-V2.6-Flashapprox. 1425088216
XiaomiMiMo/MiMo-V2.6-Pro14254n/a16
zai-org/GLM-5.3-Flash9732n/an/a

The frame count was checked directly: a 40 s video with a different caption KOD 01…KOD 40 ("CODE 01"…"CODE 40") every second, and the prompt "list the frame numbers you saw". gemma-4-31B-it listed exactly 32 evenly spaced numbers; MiMo-V2.6-Flash 16 numbers, approx. every 2.5 s; Qwen3.5-35B-A3B listed 39 of 40, although the token count implies approx. 32 frames – i.e. it fills in frames it did not see (a risk of invented content for a board with numbers).

Converted to a 45-minute lesson: one frame per approx. 84 s (gemma/Qwen/Ling, 32 frames) or per approx. 169 s (MiMo, 16 frames).

3. Audio as a separate message part

  • input_audio format: only wav and mp3. Hard limit of 600 s per part (above it, the error "Audio exceeds maximum allowed duration of 600s").
  • A 40 MB payload (base64) went through.
  • Speech quality in omni models (one clip, synthetic, clean): MiMo-V2.6-Flash garbled a rare word („bakłażan” → „baczurzan”); google/gemini-3.1-flash-lite transcribed it without error.
  • On a degraded recording (back level, 9 min, synthesis; ASR, section 2): MiMo-V2.6-Pro and -Flash: WER > 800% (loops of „w tym samym roku…” ("in the same year…")); thinkingmachines/Inkling, Inkling-Small and google/gemma-4-31B-it – the interface rejected audio input (HTTP 400) or the model was unfit for Polish transcription.
  • Omni models returned plain text only, with no word timestamps.