Papers
arxiv:2611.19244

Agentic On-Policy Distillation Implicitly Reshapes LLMs' ReAct Patterns

Published on Nov 14
· Submitted by
DeyangKong
on Nov 18
Authors:
,
,
,
,
,

Abstract

On-policy distillation can improve or degrade language-model agent interaction across domains, and selectively retaining teacher-preferred positions mitigates negative transfer at a cost to source-domain learning.

On-policy distillation (OPD) transfers specialized capabilities to language-model agents, but its cross-domain effects on interaction remain unclear. We investigate how OPD reshapes ReAct interaction patterns across mathematics, ALFWorld, and τ^2-bench, focusing on reasoning, actions, and observation acquisition. General-reasoning supervision reduces mathematical tool attempts from 70.4% to 14.4% while increasing accuracy from 16.7% to 38.0%; code-interpreter supervision improves household actions and retail queries. In contrast, ALFWorld supervision reduces τ^2-bench accuracy from 41.5% to 6.6%, accompanied by frequent tool calls that violate user-information dependencies. To address negative transfer using only source-domain training trajectories, we explore selective OPD guided by teacher preference changes before and after domain-specific RL. Retaining the top 95% of positions raises τ^2 accuracy from 11.5% to 29.9% with a source-learning trade-off.

Community

Paper author Paper submitter

On-policy distillation (OPD) transfers specialized capabilities to language-model agents, but its cross-domain effects on interaction remain unclear. We investigate how OPD reshapes ReAct interaction patterns across mathematics, ALFWorld, and τ2-bench, focusing on reasoning, actions, and observation acquisition. General-reasoning supervision reduces mathematical tool attempts from 70.4% to 14.4% while increasing accuracy from 16.7% to 38.0%; code-interpreter supervision improves household actions and retail queries. In contrast, ALFWorld supervision reduces τ2-bench accuracy from 41.5% to 6.6%, accompanied by frequent tool calls that violate user-information dependencies. To address negative transfer using only source-domain training trajectories, we explore selective OPD guided by teacher preference changes before and after domain-specific RL. Retaining the top 95% of positions raises τ2 accuracy from 11.5% to 29.9% with a source-learning trade-off.

The drop from 41.5% to 6.6% on τ^2-bench after ALFWorld supervision is the number that stuck with me. I'd like to know if the extra tool calls that break user-information dependencies show up from the first turns, or only after a few steps. Did you look at when in the trajectory the violations start?

·

Thanks for asking! 😊 We went back to the annotated trajectories and found that, of the 800 τ² rollouts from the ALFWorld-trained student, 446 contained at least one tool call that depended on information or approval the assistant had just requested. Among those 446 rollouts, the first violation occurred in the very first assistant reply in 251 cases (56.3%), and in one of the first two replies in 342 cases (76.7%).

This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend

The cross-domain contrast is useful: general-reasoning supervision can reduce unnecessary mathematical tool use while raising accuracy, whereas ALFWorld supervision drops τ²-bench accuracy from 41.5% to 6.6% with tool calls that violate user-information dependencies. That is a clear negative-transfer result on interaction patterns, not just on the source metric.

A question I would like to understand better (I have not rerun the experiments): for selective OPD, retaining the top 95% of positions raises τ² accuracy from 11.5% to 29.9% with a source-learning trade-off. How large is that source-domain cost, and did any retention threshold keep a pre-declared non-inferiority on the source while still limiting the τ² drop? I ask because a filter chosen only to recover the target may still be costly on the domain that supplied the trajectories.

— Jack Tan (@Jacktan007 )

·

Thanks for asking about the source-side trade-off. 😊 We found it encouraging that, after 20 updates, selective OPD more than doubled τ² accuracy compared with full OPD, from 11.5% to 29.9% (about a 160% relative gain), while ALFWorld accuracy went from 87.6% to 76.6% (an 11-point drop, or about 12.5% relative). What interested us most was that the τ² gain came from changing supervision on ALFWorld trajectories alone. Using the teacher’s pre-RL checkpoint as a reference to select token positions also changed a specific ReAct behavior on τ². When supervision from both domains was available, our separate multi-teacher OPD experiment reached 83.4% on ALFWorld and 68.9% on τ² using an ALFWorld teacher and a τ² teacher.

This first study was exploratory. Going forward, we’d like to see whether we can keep the gains on τ² while matching full OPD’s accuracy on ALFWorld, still using only ALFWorld trajectories for training.

The τ²-bench drop from 41.5% to 6.6% after ALFWorld supervision, with tool calls that ignore user-information dependencies, is a sharp example of a gain in one domain shipping a behaviour regression in another. Did you look at which tools got called too early (acting before asking the user), or was it spread across the tool set? A per-tool breakdown would make this easy to check before deploying a distilled agent.

(Disclosure: I work on EvalShift, an open-source tool for catching tool-routing changes between model versions: https://evalshift.dev)

Sign up or log in to comment

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.19244 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2611.19244 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.19244 in a Space README.md to link it from this page.

Collections including this paper 1