Recovering Reliable Supervision via Critique-Guided On-Policy Distillation
Abstract
CriticOPD improves on-policy distillation by using teacher critiques to restore recoverability, enabling students to revise incorrect reasoning and achieve stronger mathematical reasoning performance.
On-policy distillation (OPD) trains a student model on its own rollouts with token-level supervision from a teacher, assuming that the teacher can reach a correct answer from any prefix the student writes. We call this property the recoverability of the teacher and show it fails on incorrect rollouts: the teacher follows student errors, although it recognizes them. We propose CriticOPD, using error recognition to restore recoverability. The teacher critiques an incorrect rollout, allowing the student to rewrite its solution following the error. The resulting output, including the critique as privileged information, is scored by the teacher while the student uses the revision without the critique for training. On two Qwen3 students and six mathematical reasoning benchmarks, CriticOPD improves OPD, with the largest gains on competition problems.
Community
π€ CriticOPD: Recovering Reliable Supervision via Critique-Guided On-Policy Distillation
π Key finding: Teachers can recognize student mistakes yet still follow them when continuing a flawed solution, making OPD supervision unreliable.
βοΈ Our approach: The teacher uses the reference answer to critique incorrect student rollouts, and the student rewrites the continuation with this feedback. The teacher scores the rewritten suffix with the feedback in context, while the student learns from that suffix without seeing the feedback.
π Results: Across six math reasoning benchmarks, CriticOPD improves average accuracy over OPD by 4.57 percentage points on Qwen3-1.7B and 4.06 on Qwen3-0.6B, with the largest gains on competition problems.
π Project page Β· π» Code
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- How Should Teachers Be Prepared? RL on Student-Induced States for On-Policy Distillation (2026)
- Learning from Evolving Errors: Adaptive Iterative Repair for On-Policy Distillation (2026)
- A Privileged Teacher That Only Scores Cannot Search: Critique-Conditioned On-Policy Self-Distillation for Reasoning (2026)
- Learning from Repaired Reasoning: Root-Cause-Guided On-Policy Distillation (2026)
- Recursive Self-Improvement via On-Policy Distillation for Reasoning (2026)
- Step-Level On-Policy Distillation: Interpolating Between On-Policy Distillation and Supervised Fine-Tuning (2026)
- VISTA: Verifier-Informed Student-to-Teacher Adaptation for On-Policy Self-Distillation (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Get this paper in your agent:
hf papers read 2610.20572 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 1
Collections including this paper 0
No Collection including this paper