What Does Multi-Turn Reinforcement Learning Really Improve in LLM Agents?
Abstract
While multi-turn agentic reinforcement learning (RL) excels on long-horizon tasks, its performance gains remain highly sensitive to interaction constraints. Evaluating across sampling breadth K and interaction depth T via Pass@(K,T), we show that trajectory-level RL advantages depend on sufficient turn budgets, diminishing significantly when budgets tighten. Besides, agentic RL does not uniformly elevate single-step decision quality; rather, its strength concentrates in error recovery and long-horizon resilience. To mitigate the deployment risks caused by this budget dependence, we provide a preliminary discussion on the benefits and limitations of process supervision signals and introduce BudgetMix RL as an alternative that varies interaction budgets during training. Experiments demonstrate that BudgetMix RL achieves robust performance across diverse execution settings. Project Page: https://rethinking-agentic-rl.github.io/
Community
While multi-turn agentic reinforcement learning (RL) excels on long-horizon tasks, its performance gains remain highly sensitive to interaction constraints. Evaluating across sampling breadth K and interaction depth T via Pass@(K,T), we show that trajectory-level RL advantages depend on sufficient turn budgets, diminishing significantly when budgets tighten. Besides, agentic RL does not uniformly elevate single-step decision quality; rather, its strength concentrates in error recovery and long-horizon resilience. To mitigate the deployment risks caused by this budget dependence, we provide a preliminary discussion on the benefits and limitations of process supervision signals and introduce BudgetMix RL as an alternative that varies interaction budgets during training. Experiments demonstrate that BudgetMix RL achieves robust performance across diverse execution settings.
Really useful framing. Pass@(K,T) makes visible something that plain trajectory-level success rates hide. The BFCL row in Table 1 stood out to me: at a 25% turn budget, GRPO-Full on Qwen3-4B lands at 2.5 versus 3.0 for the base model, while BudgetMix reaches 8.5. Since BudgetMix puts the total and remaining budget in the prompt, I'm curious how it behaves when the cap isn't announced, which is common in real deployments where the limit is a timeout or a cost ceiling. Does it learn a generally more careful early policy, or does it mostly condition on the stated budget?
Thank you for your interest in our work and for raising this insightful question! In our current evaluation protocol, all models (Base, RL, and BudgetMix) are explicitly informed of the interaction budget through the same prompt. Our results therefore mainly reflect how well models adapt their strategies to a given budget, such as making more careful decisions when interactions are limited. However, as you pointed out, an important question is whether BudgetMix naturally adopts a more cautious early-stage policy even without explicit budget information. We haven't tested this setting yet, but we will consider exploring it and analyzing the resulting behaviors in future work. Thanks again for your thoughtful question and suggestion!
Get this paper in your agent:
hf papers read 2611.06483 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper