# CPTU-Bench

Checks how well AI models really understand Polish: reading between the lines (sarcasm, hidden meaning, idioms) and answering tricky questions with logic traps, ambiguity or absurd premises. It shows whether a model grasps Polish beyond the literal words.

Category: polish. Type: leaderboard. Status: active.
Metric: Average score on a 0–5 scale across four parts (sentiment, language understanding, phraseology, tricky questions) (higher is better).
Source: SpeakLeash — https://huggingface.co/spaces/speakleash/cptu_bench/resolve/main/data.json, results as of 2026-08-11, retrieved 2026-11-17, license: MIT.

| Rank | Model | Params | Average | Sentiment | Language understanding | Phraseology | Tricky questions |
|---|---|---|---|---|---|---|---|
| 1 | meta-models/Muse-Glimmer-30B reasoning (API, FP8) | 30B | 4.42 | 4.52 | 4.39 | 4.23 | 4.52 |
| 2 | Qwen/Qwen3.6-27B reasoning (API, FP8) | 27B | 4.41 | 4.58 | 4.21 | 4.21 | 4.64 |
| 3 | Qwen/Qwen3.5-122B-A10B reasoning (API, FP8) | 122B | 4.36 | 4.36 | 4.18 | 4.19 | 4.72 |
| 4 | Qwen/Qwen3.5-27B thinking (API) | 27B | 4.34 | 4.42 | 4.21 | 4.11 | 4.61 |
| 5 | google/gemma-4-31B-it non-reasoning (API, FP8) | 31B | 4.32 | 4.58 | 4.46 | 4.07 | 4.19 |
| 6 | google/gemma-4-31B-it reasoning (API, FP8) | 31B | 4.31 | 4.58 | 4.38 | 3.77 | 4.52 |
| 7 | Qwen/Qwen3.6-27B non-reasoning (API, FP8) | 27B | 4.29 | 4.33 | 4.08 | 4.40 | 4.36 |
| 8 | gemini-2.0-flash-001 |  | 4.29 | 4.52 | 4.32 | 4.34 | 3.99 |
| 9 | Qwen/Qwen3.5-27B non-thinking (API) | 27B | 4.27 | 4.29 | 4.17 | 4.20 | 4.43 |
| 10 | Qwen/Qwen3.5-122B-A10B non-reasoning (API, FP8) | 122B | 4.26 | 4.33 | 4.24 | 3.94 | 4.53 |
| 11 | google/gemma-4-26B-A4B-it non-reasoning (API, FP8) | 26B | 4.23 | 4.39 | 4.00 | 4.21 | 4.30 |
| 12 | Qwen/Qwen3.5-35B-A3B thinking (API) | 35B | 4.22 | 4.10 | 3.94 | 4.15 | 4.70 |
| 13 | google/gemma-4-26B-A4B-it reasoning (API, FP8) | 26B | 4.21 | 4.39 | 4.08 | 3.91 | 4.47 |
| 14 | Qwen/Qwen3.5-35B-A3B non-thinking (API) | 35B | 4.18 | 4.23 | 4.05 | 4.23 | 4.19 |
| 15 | deepseek-ai/DeepSeek-V3.2 (API) | 685B | 4.14 | 4.46 | 4.36 | 3.54 | 4.20 |
| 16 | deepseek-ai/DeepSeek-R1 (API) | 685B | 4.14 | 4.49 | 4.34 | 3.60 | 4.12 |
| 17 | gemini-2.0-flash-lite-001 |  | 4.09 | 4.23 | 4.05 | 4.24 | 3.85 |
| 18 | deepseek-ai/DeepSeek-V3-0324 (API) | 685B | 4.03 | 4.36 | 4.20 | 3.54 | 4.02 |
| 19 | deepseek-ai/DeepSeek-V3.1 (API) | 685B | 4.03 | 4.42 | 4.33 | 3.48 | 3.87 |
| 20 | deepseek-ai/DeepSeek-V3 (API) | 685B | 4.02 | 4.36 | 4.22 | 3.52 | 3.99 |
| 21 | mistralai/Mistral-Large-Instruct-2411 | 123B | 4.00 | 4.33 | 3.98 | 3.99 | 3.72 |
| 22 | moonshotai/Kimi-K2-Instruct-0905 (API) | 1000B | 3.98 | 4.39 | 4.18 | 3.43 | 3.93 |
| 23 | Qwen/Qwen2.5-72B-Instruct | 72.7B | 3.95 | 4.08 | 3.97 | 3.93 | 3.81 |
| 24 | mistralai/Mistral-Large-Instruct-2407 | 123B | 3.93 | 4.23 | 4.00 | 3.86 | 3.65 |
| 25 | meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8 (API) | 402B | 3.93 | 4.39 | 4.11 | 3.48 | 3.76 |

Notes: Per-category scores copied from the Space's data.json; 'Average' is the plain mean of the four categories, computed exactly as the leaderboard app does (app.py) and shown to 2 decimals like the leaderboard. as_of is the Space's last commit date. License is the Space repository's declared license (MIT). 102 models in total; top 25 shown.
