Post
90
New paper: Length Is Cheap, Competition Is Not 📄
Why do LLMs fail on long inputs? For retrieval, our answer is: not because of length.
• 32K tokens of neutral text: Qwen2.5-1.5B keeps 0.97–1.00 accuracy
• Same positions, statements in the evidence format: 0.11–0.21
• One rival value for the queried key: loss ×9 to ×110
The attention mass left on the evidence predicts the loss (up to 58% of variance, 126K trials). Log length: at most 8%.
Paper: Length Is Cheap, Competition Is Not: Attention Predicts Context Rot in Retrieval (2611.04169)
Data (all 126K trials): yliusci/length-is-cheap
Feedback very welcome, especially on larger models.
Why do LLMs fail on long inputs? For retrieval, our answer is: not because of length.
• 32K tokens of neutral text: Qwen2.5-1.5B keeps 0.97–1.00 accuracy
• Same positions, statements in the evidence format: 0.11–0.21
• One rival value for the queried key: loss ×9 to ×110
The attention mass left on the evidence predicts the loss (up to 58% of variance, 126K trials). Log length: at most 8%.
Paper: Length Is Cheap, Competition Is Not: Attention Predicts Context Rot in Retrieval (2611.04169)
Data (all 126K trials): yliusci/length-is-cheap
Feedback very welcome, especially on larger models.
- 1 reply