Why We Built Our Own Uzbek Benchmark

Every serious model release ships with benchmark scores: MMLU, GSM8K, HumanEval. Look closely and you'll notice what they have in common — they are almost entirely in English. For Uzbek, the public evaluation landscape is close to empty. A handful of machine-translated test sets exist, and machine translation is precisely the thing that breaks on Uzbek, so the tests inherit the errors they are supposed to detect.

When we started training Tomaris, we had no honest way to answer the most basic question: is this version better than the last one? So before scaling up training, we built the measuring stick.

The 300-prompt eval

Our benchmark is 300 human-written prompts across four tracks, each targeting a failure mode we observed in global models:

• Fluency & linguistics — agglutinative morphology, vowel harmony edge cases, register shifts between formal and spoken Uzbek.
• Cultural knowledge — history, literature, customs, and everyday context that never appears in web-crawled English data.
• Morphology under pressure — long derivational chains where one wrong suffix changes the meaning of the sentence.
• Math & logic — 105 Olympiad-grade problems sourced from Uzbekistan Olympiad archives, written natively in Uzbek.

The Olympiad track deserves explanation. Math problems written natively in Uzbek — with Uzbek phrasing, Uzbek names, Uzbek problem conventions — cannot be solved by pattern-matching against English training data. A model must actually parse the Uzbek and reason. That makes them the hardest and most honest part of the eval.

How we use it

Every Tomaris checkpoint runs the full 300 prompts before release, and native speakers grade the outputs. We don't publish leaderboard-style victory numbers, because grading long-form Uzbek is partly subjective and we would rather earn trust slowly than lose it fast. What we can say: the benchmark is why Tomaris improves measurably between versions instead of by feel.

A benchmark nobody else can pass isn't a moat. A benchmark nobody else even has — that's a moat.

We plan to open parts of the benchmark to the research community once the grading rubric is stable. If you work on low-resource language evaluation, we would genuinely like to talk.