Thanks to Kyle and the AllenAI team for this benchmark! The finding that agentic search and complex evaluators underperform simpler baselines strongly resonates with what I observed in my own research: https://coronring.github.io/blog/stop-taking-ai-advice-on-where-to-submit/
Earlier this month, I ran a complementary study asking current models (GPT-6 Luna and Claude Opus 5.5) where to submit 132 full, blinded AI papers from the 2026 cycle (NeurIPS, EMNLP, COLM, Findings, and workshops) strictly after the model knowledge cutoff date: Stop taking AI advice on where to submit your paper.
While Novelty-Bench examines the idea-level novelty judge, evaluating full manuscripts revealed the exact same underlying failure modes:
- "Careful review" degraded performance: Prompting the model to write a comprehensive program-committee review and estimate acceptance odds first dropped tier accuracy from 39% to 30%, mirroring your finding that heavier reasoning pipelines (like AI Scientist) fell behind simple prompted baselines.
- Superficial heuristics over substance: Just as your judges reacted to generator formatting styles, our evaluator followed venue topic stereotypes (NLP to EMNLP, vision to NeurIPS) and length. In fact, ranking papers by simple word count matched or outperformed the model's predicted acceptance odds across multiple tiers.
- 0 of 29 top contributions recognized: The model never once recommended aiming for an oral or spotlight, treating NeurIPS workshop papers and actual oral presentations with almost identical estimated probabilities (~2.5% vs 2.7%).
- Completely uncalibrated confidence: Confidence was virtually indistinguishable when the model was right (0.42) versus wrong (0.40).
Your work provides the benchmark tools needed to quantify this at the ideation stage, and our results suggest the problem persists all the way downstream to full paper evaluation. Very glad to see rigorous attention brought to this issue and love to chat on it!
