At COLM 2026, the paper 'Cracks in the Foundation' from Ai2 and others systematically challenges several standard Transformer designs: QK normalization, GQA, sliding window attention, and pretraining context length. The researchers selected different values from Llama 2, Llama 3, Qwen 3, and Olmo 3, training 26 models with 7B-8B parameters, collectively called 'OlmPool', keeping data, tokenizer, and context extension scheme consistent, only varying architecture.
These models were first pretrained on 140B tokens, then fine-tuned for long context with 10B tokens, consuming over 170,000 GPU hours in total. On the HELMET 32K benchmark, the highest score was 56.4 and the lowest 29.9, a relative gap of 47%. Adjusting a single switch (such as pretraining context length or sliding window) on average affects only one or two points, but combined effects are significant: SWA and GQA together drop an average of 9 points, exceeding the sum of their individual effects; the worst combination (GQA+SWA+per-head QK normalization) also drops more than linearly.
A counterintuitive finding is that QK normalization (originally added for training stability) when removed and replaced with pre-normalization on Olmo 3 improves HELMET scores by 6 points, but the same change on Llama 3 decreases by 3.8 points, indicating that the same design can have opposite effects on different models. Additionally, the paper finds that 'attention sinking' (where models concentrate a large amount of attention on tokens at the beginning of the context) is positively correlated with long-context performance in OlmoPool models, and QK normalization weakens this sinking, which may explain its negative impact.