benchmarks.site

New to this?

Glossary

The words you'll meet on this site, explained without jargon.

Accuracy

Share of answers that were correct. Higher is better.

ASR

Speech-to-text: turning spoken audio into written text.

Automatic speech recognition, also called transcription.

AUC

How well a classifier separates two groups: 1.0 is perfect, 0.5 is a coin flip.

Benchmark

A standard test used to compare AI models on the same task.

Like an exam that every model takes under the same rules, so the scores can be compared. A benchmark only tells you about the task it tests — a model that wins at coding may be average at reading handwriting.

CER

Character error rate: share of characters read wrong. Lower is better.

Like WER but per letter. 0.02 (or 2%) means about 2 wrong characters in 100.

Contamination

When test questions leaked into a model's training data, inflating its score.

DER

Diarization error rate: share of time given to the wrong speaker. Lower is better.

Diarization

Working out who is speaking when in a recording.

Elo

A rating from head-to-head votes, like in chess. Higher is better.

Used by arena leaderboards: people compare two anonymous answers and pick the better one. A 100-point gap means the higher model wins roughly 64% of matchups.

Embeddings

Turning text into lists of numbers so that similar meanings end up close together.

This is what powers search and 'find similar' features: the question and the documents are embedded, and the closest documents are returned.

False positive

A false alarm: something harmless flagged as a problem. Lower is better.

Latency

How long you wait for an answer, in seconds. Lower is better.

Often split into time to first token (when the answer starts appearing) and total time (when it is finished).

Leaderboard

A ranked table of models' scores on a benchmark.

Usually maintained by the benchmark's authors. Check its date: models change every few weeks.

LLM

Large language model — the kind of AI behind chat assistants.

A model trained on huge amounts of text that can write, answer questions, summarize and follow instructions.

OCR

Reading text from images (photos, scans).

Optical character recognition. Here usually done by vision-language models that look at the photo and write out the text.

Open-weights model

A model whose files are published, so anyone can run it on their own hardware.

pass@1

Share of tasks solved on the first try. Higher is better.

Platform

The service you call the model through (e.g. an API provider). The same model can behave differently on different platforms.

Precision

Share of the things flagged that were actually correct. Higher is better.

Reasoning

Hidden 'thinking' a model does before answering.

Reasoning models write internal steps (reasoning tokens) before the answer. It can improve hard tasks but costs time and money, because those tokens are billed too.

Recall

Share of the things you were looking for that were actually found. Higher is better.

Saturated

A benchmark where top models score near the maximum, so it no longer separates them well.

Token

A piece of text (roughly ¾ of a word) — the unit models read, write and bill by.

TTS

Text-to-speech: turning text into spoken audio.

WER

Word error rate: share of words transcribed wrong. Lower is better.

Counts wrong, missing and extra words against the correct text. 10% means roughly 1 word in 10 is wrong. It can exceed 100% if the model adds a lot of text that wasn't said.