# Omni models: video and audio input

> On DeepInfra, `video_url` input delivers only image frames to the model (audio was dropped for 6 of 6 models), and models see a fixed number of frames (32 or 16 where measured) regardless of video length; audio has to be sent as a separate `input_audio` part (wav/mp3 only, max 600 s per part).

Area: Speech. Page: https://benchmarks.site/omni-video-audio/. Methodology and caveats: https://benchmarks.site/methodology/#speech-vision-text

# "Omni" models: video and audio as input

Platform: DeepInfra. Measured on 2026-09-23 against the live API, supplemented with results from 2026-09-29. All numbers come from API responses (`usage`), not from documentation. Metric definitions and caveats: [overview](/methodology/#speech-vision-text).

## Catalog (2026-09-23)

From the DeepInfra model list: 232 models not marked as deprecated, including 17 with a video-input tag, 11 with an audio-input tag and 8 native ASR models. The tag is binding: a model without the tag responds with HTTP 405 ("does not accept video input"); e.g. `google/gemini-3.1-flash-lite` does not accept video (audio only). All Gemini models on DeepInfra are forwarded to Google (partner flag); `google/gemma-4-31B-it-Ultra` is also a partner model (despite open weights), while the plain `gemma-4-31B-it` and `-turbo` are hosted by DeepInfra.

## 1. Does the model hear the audio from a video file?

- Test clip: 7.7 s, a slide showing `x + 7 = 12` / `Zadanie domowe: str. 42` ("Homework: p. 42"), a speech track with a "secret word" („bakłażan” ("eggplant")). Prompt: transcribe the board and quote what can be heard; if nothing can be heard, write `BRAK DZWIEKU` ("NO SOUND"). 1 clip, 1 trial per model.

| model | board read | audio from `video_url` |
|---|---|---|
| google/gemma-4-31B-it (video + audio) | yes | no audio |
| XiaomiMiMo/MiMo-V2.6-Flash (video + audio) | yes | no audio |
| XiaomiMiMo/MiMo-V2.6-Pro (video + audio) | yes | no audio |
| Qwen/Qwen3.5-35B-A3B | yes | no audio |
| inclusionAI/Ling-3.0-flash-VL | yes | no audio |
| zai-org/GLM-5.3-Flash | yes | no audio |

Conclusion: **on DeepInfra, `video_url` input delivers only image frames to the model; the audio is discarded** (six models out of six). Evidence: the same MiMo-V2.6-Flash transcribed the same utterance word for word when it received it as a separate audio part, and with video and audio in one message it responded to both.

## 2. How many frames does the model see?

The number of input tokens for video was **identical** for videos of 30 s, 60 s, 300 s, 720 s and 2700 s, and for 1280×720 and 640×360 – a fixed number of frames regardless of length. No video length limit was found (a 2700 s video went through), nor any way to set the number of frames.

| model | tokens per video (any length) | tokens per 1 full image | frames |
|---|---|---|---|
| google/gemma-4-31B-it | 2421 | 266 | 32 |
| inclusionAI/Ling-3.0-flash-VL | approx. 4400 | 883 | approx. 32 |
| Qwen/Qwen3.5-35B-A3B | approx. 11720 | 883 | approx. 32 |
| XiaomiMiMo/MiMo-V2.6-Flash | approx. 14250 | 882 | 16 |
| XiaomiMiMo/MiMo-V2.6-Pro | 14254 | n/a | 16 |
| zai-org/GLM-5.3-Flash | 9732 | n/a | n/a |

The frame count was checked directly: a 40 s video with a different caption `KOD 01`…`KOD 40` ("CODE 01"…"CODE 40") every second, and the prompt "list the frame numbers you saw". gemma-4-31B-it listed exactly 32 evenly spaced numbers; MiMo-V2.6-Flash 16 numbers, approx. every 2.5 s; **Qwen3.5-35B-A3B listed 39 of 40, although the token count implies approx. 32 frames** – i.e. it fills in frames it did not see (a risk of invented content for a board with numbers).

Converted to a 45-minute lesson: one frame per approx. 84 s (gemma/Qwen/Ling, 32 frames) or per approx. 169 s (MiMo, 16 frames).

## 3. Audio as a separate message part

- `input_audio` format: only `wav` and `mp3`. Hard limit of 600 s per part (above it, the error "Audio exceeds maximum allowed duration of 600s").
- A 40 MB payload (base64) went through.
- Speech quality in omni models (one clip, synthetic, clean): MiMo-V2.6-Flash garbled a rare word („bakłażan” → „baczurzan”); `google/gemini-3.1-flash-lite` transcribed it without error.
- On a degraded recording (`back` level, 9 min, synthesis; [ASR](/asr-polish/), section 2): MiMo-V2.6-Pro and -Flash: WER > 800% (loops of „w tym samym roku…” ("in the same year…")); `thinkingmachines/Inkling`, `Inkling-Small` and `google/gemma-4-31B-it` – the interface rejected audio input (HTTP 400) or the model was unfit for Polish transcription.
- Omni models returned plain text only, with no word timestamps.
