benchmarks.site

Platforms

Platforms: availability, latency and limits

What it is actually like to use AI model hosting services: delays, errors, queues and hidden limits.

Key finding

Always check the `model` field in responses (DeepInfra silently redirects retired models to replacements), expect load-dependent latency and queues on DeepInfra, and note Soniox's default limit of 10 parallel streams per organization; OpenAI live transcription provides no end-of-utterance marker or word timestamps.

How to read the numbers

Observations with dates; latency in seconds — lower is better.

Hover a column name for its meaning, or see the glossary.

How we tested & caveats · Markdown · CSV

Observations about the platforms (DeepInfra, Soniox, OpenAI), collected while measuring models (September–November 2026). This is the state as of the given dates; platforms change, and load affects latency. Metric definitions and caveats: overview.

DeepInfra

12 rows · 3 columnsCSV
Queue/overload of Qwen3-ASR-1.7B: for approx. 25 min 429 Model busy responses, then for approx. half an hour a queue of 114–186 s2026-09-29a single episode
Whisper and other models: latency depends on load, e.g. whisper-large-v3 up to 23 s for a 48 s clip; Qwen3-ASR 4–22 s for 10 min of audio and 15–27 s for 46 min2026-09-22
Qwen3-ASR, 20 s chunk: 50–77 s under platform load (Nemotron 1.4–3.5 s)2026-11-05normal mode and priority gave similar times
flex tier (lowest priority): on a short query, time to first response for DeepSeek-V4-Pro went from 1.3 s to 13–20 s, for DeepSeek-V4.1-Flash from 0.8–1.0 s to 3.6–4.8 s; according to a later assessment, flex latency was 8–40 s higher than in normal mode even without a queue2026-09-30speech models (Qwen3-ASR, Whisper, Nemotron) accepted this field with no effect
Chat models under load: Kimi-K3 HTTP 429 engine_overloaded with 4 retries over approx. 30 s; GLM-5.3-Flash 75–206 s to first token and 180 s timeouts; Kimi-K2.6 once 115 s to first token; DeepSeek-V4.1-Flash one 180 s timeout in approx. 25 calls2026-09-22
Retired models are silently redirected to a replacement (replaced_by), so "DeepSeek-OCR" and "olmOCR" responded as google/gemma-4-31B-it, small Qwen3.5 models (0.8B–4B) as Qwen3.5-9B, and Phi-4-multimodal as gemma-3-12b2026-10-06results require checking the model field in the response
The cost field in the native ASR interface has inconsistent units: cents for Whisper and Voxtral, dollars for Qwen and Nemotron2026-09-22
Images: MIME image/jpg and image/heic – HTTP 422 with a misleading message; HEIC bytes labeled as jpeg – HTTP 400 on models hosted by DeepInfra, while Gemini and Seed accept them; the documented 20 MB limit is not enforced (a 21.8 MB PNG was accepted); ByteDance Seed-2.0-mini has its own limits of 10 MiB and 36 Mpx (reported as HTTP 500)2026-09-22
Video: the video_url part passes only the image to the model (see video-audio); audio input has a limit of 600 s per part2026-09-23
Prompt caching: a repeated query with the same image and prompt is sometimes billed cheaper (automatic cache hits, varying between models)2026-09-22
Claude (Sonnet, Opus) via DeepInfra: no reported cache hits2026-11-06see LLM text, section 6
Gemini models are forwarded to Google (partner flag); gemma-4-31B-it-Ultra is also a partner model2026-09-23

Soniox (live and file speech recognition)

  • Limit of 10 parallel streams per organization (default); with 12 streams, 2 received 429 (2026-11-06).
  • With a congested uplink (12 streams + parallel file uploads), the first result took up to 7.7 s and streams were dropped (6 at once at second 46, including 3 with 408 Request timeout and 3 without an error; one at second 63 without an error). Without other traffic, 30 of 30 streams ran to completion. The cause was not proven (the client-side link is the most likely).
  • The final result (is_final) in continuous speech arrived approx. 5.7 s after the word; the partial results are the useful ones.
  • Latin is not supported (a language hint including Latin gives HTTP 400 for the whole request).
  • Polish recordings, two model releases: stt-rt-v3 is an alias of stt-rt-v5; the newest stt-rt-v6 gave approx. 1 point lower WER, but the first result approx. 300 ms later (ASR, section 5).

OpenAI (transcription models)

  • gpt-live-transcribe and gpt-realtime-whisper in live mode send text deltas without an end-of-utterance marker or word timestamps, and reject server-side voice activity detection (2026-11-06).
  • The gpt-4o-* models are being shut down by OpenAI on 2027-02-26.
  • GPT-6 (Responses API): the temperature parameter causes error 400; changing the reasoning level between queries invalidates the cache (2026-11-06).