Woodpecker Distillation: Weak Models Diagnose Reasoning Bugs in Strong Models

Large language models often fail on reasoning tasks despite possessing the underlying capability, and the paper argues these failures frequently stem from localized reasoning bugs in intermediate steps rather than global incompetence. Such bugs are repairable: inserting a short patch generated by a weak probe model after the same strong-model reasoning prefix can redirect the trajectory toward a correct solution. However, directly fine-tuning on weak patches or repaired trajectories does not reliably internalize the corrective effect, suggesting the useful signal lies in how the intervention reshapes the model's future reasoning distribution rather than in the intervention text itself.

Woodpecker Distillation leverages this insight by contrasting successful and unsuccessful weak-model patches at the same prefix, constructing a corrective teacher distribution from their induced future token predictions, and distilling this signal into the strong model. Experiments on mathematical reasoning benchmarks show that the method consistently improves strong-model performance and outperforms direct imitation baselines. This work provides a new perspective on weak-to-strong learning and offers a practical approach to improving reasoning capabilities without requiring stronger external supervision.