benchmarks.site

Embeddings

Embeddings: retrieval accuracy in Polish

Embeddings power search: they turn text into numbers so similar texts can be found. Which models find the right Polish note?

Key finding

On a small Polish note set, the Qwen3-Embedding models (0.6B, 4B, 8B) ranked relevant notes above lookalike distractors, with Qwen3-Embedding-0.6B giving the best margin and 10/10 queries with an instruction prefix; bge-m3, multilingual-e5-large and embeddinggemma-300m let unrelated notes outrank a relevant one.

How to read the numbers

Accuracy — higher is better. Margin — how clearly the right note beats look-alikes; higher is better.

Hover a column name for its meaning, or see the glossary.

How we tested & caveats · Markdown · CSV

Platform: DeepInfra. Measurement: 2026-09-22. Description of metrics and caveats: project overview.

Data

  • 12 Polish note documents, including deliberately misleading, off-topic ones: „równania reakcji” ("reaction equations"), „równania Maxwella” ("Maxwell's equations"), „układy równań liniowych” ("systems of linear equations"), „kwadratura koła” ("squaring the circle").
  • Graded test for the query „równania kwadratowe” ("quadratic equations"): a point score (0–1 scale; the exact definition is not given in the source) and a margin = lowest score of a relevant note minus highest score of an irrelevant note (a negative value = an irrelevant note outranked a relevant one).
  • 10-query test: point score 0–1 (1.00 = full accuracy); for bge-m3 the source gives 0.95 and 9 of 10 queries with a relevant note in first place.
  • Small sample: 12 documents, 10 queries, one call per configuration.
  • "With instruction" = the query is prefixed with a short task instruction (recommended for the Qwen3-Embedding family; documents without a prefix).
6 rows · 5 columnsCSV
BAAI/bge-m310241.5–1.6 s0.632 / −0.0630.95 (9/10 in first place)
Qwen/Qwen3-Embedding-0.6B10241.5–3 s (one cold 26 s)0.953 / +0.179 without instruction; 0.935 / +0.187 with instruction0.925 without instruction; 1.00 with instruction
Qwen/Qwen3-Embedding-4B25601.5–5.8 s0.953 / +0.078; with instruction margin +0.127; at 1024 dimensions 0.888 / +0.1041.00
Qwen/Qwen3-Embedding-8B409610–31 s0.953 / +0.048; with instruction margin +0.0971.00
multilingual-e5-large1024 (512-token limit)1.6–29 s0.751 / −0.030 (similarities clustered in the 0.73–0.88 range)1.00
embeddinggemma-300m7680.9–1.6 s0.817 / −0.0361.00

Observations:

  • bge-m3 ranked the note with Viète's formulas below three unrelated notes that only share the word „równania” ("equations") or „kwadrat” ("square").
  • Qwen3-Embedding-0.6B had the best margin (+0.19) on the graded test and 10/10 with the instruction.
  • Qwen3-Embedding-4B honors the dimension-reduction parameter (down to 1024).
  • Vectors from different models are not interchangeable.