On-policy distillation: why distillation quality gates routing

Distillation used to mean copying. Freeze a teacher, collect its outputs on a static corpus, train a student to imitate. On-policy distillation (OPD) breaks that frame: the student is trained on states induced by its own current policy. It generates rollouts, a teacher or self-teacher scores them, and the student updates on its own mistakes instead of the teacher's curated ones.

That shift made OPD one of the most active post-training lines of 2026. It is also why I care: I route real agent traffic to small models, and routing quality is a distillation question.

Field map

OPD is best read as a feedback-to-update problem, not a single loss family. The survey below organizes the field into two routes: direct distributional losses, which push the student's distribution toward the teacher's, and policy-gradient-style log-ratio updates, which weight sampled actions by teacher-student agreement. The variables that decide success are state compatibility, support construction, temporal credit, and gating. KL direction and teacher access matter less than those.

The second axis is teacher access. Logit-level distillation gets dense signal but requires teacher internals. Black-box distillation works from text outputs alone, which is the only option for proprietary teachers. Everything I route against lives in the black-box half.

The survey also covers verifier- or outcome-guided hybrids, industrial reports, framework implementations, failure modes, and stabilization recipes, under explicit evidence boundaries. That last habit is worth copying: the paper is careful about which claims are demonstrated and which are hypotheses. For where this sits in the models pillar, see foundation-models-field-map.

Close-read: three papers

The survey: OPD as a feedback-to-update problem

A Formula-Driven Survey and Research Agenda for On-Policy Distillation (Zhang, 2026) is the map. It separates two mechanisms that sampled-token OPD discussions usually conflate. Temporal credit asks how teacher-student log-ratio returns should weight sampled actions across a rollout. Vocabulary routing asks where probability mass should move when negative feedback suppresses a sampled token. Estimator choices (immediate, return-to-go, discounted, baseline-corrected) sit on a bias boundary, and the survey motivates GAE-OPD as a value-based hypothesis for log-ratio returns, plus counterfactual routing toward teacher-supported, student-reachable alternatives. The closing sections are the part I read twice: open problems and a reporting checklist, both mapped back onto the same feedback-to-update variables.

Black-box OPD: the discriminator as reward model

Black-box On-Policy Distillation of Large Language Models (Ye et al., 2025) is the most interesting black-box result I have seen. Generative Adversarial Distillation (GAD) frames the student as a generator and trains a discriminator to tell student responses from teacher responses, a minimax game where the discriminator acts as an on-policy reward model that co-evolves with the student. In their experiments, Qwen2.5-14B-Instruct trained with GAD became comparable to its teacher, GPT-5-Chat, on LMSYS-Chat automatic evaluation, consistently beating sequence-level knowledge distillation.

The adversarial setup is exactly what a routing engineer should watch. An adapting reward model resists hacking, but it also makes training dynamics harder to reason about, because the student trains against a moving target. And the whole framing exists for proprietary teachers: no logits, no parameters, just text outputs, which is the constraint every real routing pipeline operates under.

The illusion of certainty

The Illusion of Certainty: Decoupling Capability and Calibration in On-Policy Distillation (Zhang et al., 2026) finds a scaling law of miscalibration: OPD reliably improves task accuracy while trapping models in severe overconfidence. The cause is an information mismatch. Teacher supervision is formed under privileged context available during training, but the deployed model must report confidence using deployment-time information only. Teacher-conditioned success is not a valid target for deployment-time confidence; helpful privileged context induces entropy collapse and a systematic optimism bias.

Their fix, CaOPD, estimates empirical confidence from model rollouts, replaces self-reported confidence with that student-grounded target, and distills the revised response through the same self-distillation pipeline. It reaches Pareto-optimal calibration while keeping capability, and generalizes under out-of-distribution and continual learning settings.

This is the paper I would force every routing team to read. Accuracy without calibration is worse than no distillation, because the failures are silent.

My position: distillation quality gates routing

My position is that routing decisions are distillation decisions. When a router sends a task to a small model, it bets on two things: the model can do the task, and the model knows when it cannot. OPD's failure modes attack both bets. Miscalibration means the small model sounds certain while wrong, which is the worst case for a router that escalates on low confidence. The failure is directional: overconfidence produces silent wrong answers, while undercapability just wastes frontier API spend and latency. Only the second one shows up in your logs.

The build that follows from this: distillation as a capability gate inside the routing pipeline, with held-out exams per task cluster, calibration measured on the served model rather than on training checkpoints — concretely, expected calibration error over the model's verbalized confidence against its empirical rollout confidence, the same student-grounded target CaOPD distills toward — and a fallback path that is exercised on schedule, not only on incident day. The OPD reading list is the spec for that gate.

Where it connects: routing at PiPlan.ai

At PiPlan.ai the dynamic routing layer runs routine agent steps on a self-hosted vLLM node and escalates complex planning to frontier APIs. VerifierForge trains small models to take over expensive task clusters. Both bets come down to one question: does the small model actually know what it is doing, and does it know when it does not? That is a distillation quality question, and it is why the OPD work above is not academic to me. See verifier-engineering for the reward side of the same loop.

Sources linked in this post were fetched and verified.