Post
87
New paper: Length Is Cheap, Competition Is Not š
Why do LLMs fail on long inputs? For retrieval, our answer is: not because of length.
ā¢ 32K tokens of neutral text: Qwen2.5-1.5B keeps 0.97ā1.00 accuracy
ā¢ Same positions, statements in the evidence format: 0.11ā0.21
ā¢ One rival value for the queried key: loss Ć9 to Ć110
The attention mass left on the evidence predicts the loss (up to 58% of variance, 126K trials). Log length: at most 8%.
Paper: Length Is Cheap, Competition Is Not: Attention Predicts Context Rot in Retrieval (2611.04169)
Data (all 126K trials): yliusci/length-is-cheap
Feedback very welcome, especially on larger models.
Why do LLMs fail on long inputs? For retrieval, our answer is: not because of length.
ā¢ 32K tokens of neutral text: Qwen2.5-1.5B keeps 0.97ā1.00 accuracy
ā¢ Same positions, statements in the evidence format: 0.11ā0.21
ā¢ One rival value for the queried key: loss Ć9 to Ć110
The attention mass left on the evidence predicts the loss (up to 58% of variance, 126K trials). Log length: at most 8%.
Paper: Length Is Cheap, Competition Is Not: Attention Predicts Context Rot in Retrieval (2611.04169)
Data (all 126K trials): yliusci/length-is-cheap
Feedback very welcome, especially on larger models.
