yliusci/length-is-cheap
Viewer ā¢ Updated ā¢ 197k ā¢ 163
Yinuo Liu
yliusci
AI & ML interests
Raw Mathematics
Recent Activity
posted an update 6 days ago
New paper: Length Is Cheap, Competition Is Not š
Why do LLMs fail on long inputs? For retrieval, our answer is: not because of length.
ā¢ 32K tokens of neutral text: Qwen2.5-1.5B keeps 0.97ā1.00 accuracy
ā¢ Same positions, statements in the evidence format: 0.11ā0.21
ā¢ One rival value for the queried key: loss Ć9 to Ć110
The attention mass left on the evidence predicts the loss (up to 58% of variance, 126K trials). Log length: at most 8%.
Paper: https://huggingface.co/papers/2611.04169
Data (all 126K trials): https://huggingface.co/datasets/yliusci/length-is-cheap
Feedback very welcome, especially on larger models. updated a dataset 6 days ago
yliusci/length-is-cheap submitted a paper 6 days ago
Length Is Cheap, Competition Is Not: Attention Predicts Context Rot in RetrievalOrganizations
None yet