benchmarks.site

Public leaderboard · Hallucination & factuality

Vectara Hallucination Leaderboard

Asks AI models to summarize short documents and uses an automated checker to count how often the summary adds things that are not in the original text. It shows how likely a model is to make things up when it should just be faithful to a source.

Leaderboard

Top 3 right now

  1. 1stantgroup/finix_s1_32b1.8 %
  2. 2ndopenai/gpt-5.4-nano-2026-03-173.1 %
  3. 3rdgoogle/gemini-2.5-flash-lite3.3 %

What it measures

Hallucination rate when models summarize short documents, judged by Vectara's HHEM classifier.

How to read the numbers

Share of summaries containing information not supported by the source document (hallucination rate, %) — lower is better.

Results

25 rows · lower is better (Hallucination Rate)CSV
1antgroup/finix_s1_32b1.8 %98.2 %99.5 %172.4
2openai/gpt-5.4-nano-2026-03-173.1 %96.9 %100.0 %144.4
3google/gemini-2.5-flash-lite3.3 %96.7 %99.5 %95.7
4microsoft/Phi-43.7 %96.3 %80.7 %120.9
5meta-llama/Llama-3.3-70B-Instruct-Turbo4.1 %95.9 %99.5 %64.6
6deepseek-ai/DeepSeek-V4.1-Flash4.2 %95.8 %99.9 %108.2
7snowflake/snowflake-arctic-instruct4.3 %95.7 %62.7 %81.4
8google/gemma-3-12b-it4.4 %95.6 %97.4 %89.7
9mistralai/mistral-large-24114.5 %95.5 %99.9 %85.0
10qwen/qwen3-8b4.8 %95.2 %99.9 %83.6
11amazon/nova-pro-v1:05.1 %94.9 %99.3 %66.2
12amazon/nova-2-lite-v1:05.1 %94.9 %99.6 %94.1
13mistralai/mistral-small-25015.1 %94.9 %97.9 %98.8
14ibm-granite/granite-4.0-h-small5.2 %94.8 %100.0 %107.4
15google/gemma-4-26b-a4b-it5.2 %94.8 %99.8 %67.1
16ai21labs/jamba-mini-25.3 %94.7 %99.6 %109.4
17deepseek-ai/DeepSeek-V3.2-Exp5.3 %94.7 %96.6 %64.6
18qwen/qwen3-14b5.4 %94.6 %99.9 %111.1
19amazon/nova-micro-v1:05.5 %94.5 %100.0 %100.0
20deepseek-ai/DeepSeek-V3.15.5 %94.5 %94.5 %63.7
21openai/gpt-5.4-mini-2026-03-175.5 %94.5 %100.0 %54.7
22openai/gpt-4.1-2025-04-145.6 %94.4 %99.9 %91.7
23qwen/qwen3-4b5.7 %94.3 %99.9 %104.7
24xai-org/grok-35.8 %94.2 %93.0 %95.9
25qwen/qwen3-32b5.9 %94.1 %99.9 %115.8

Note: Top 25 by lowest hallucination rate, in README order. Judged by Vectara's own HHEM classifier on its current (new) document set; covers summarization faithfulness only. Answer rate = share of documents the model agreed to summarize.

Source: Vectara · github.com · results as of 2026-10-22 · retrieved 2026-11-17 · license: Apache-2.0 (repository license)

Markdown · More Hallucination & factuality