Removing 99.95% of Training Signals Works Better: Extreme Sparse Supervision Reshapes LLM Post-Training

When a large model solves a math problem, it often has to generate a reasoning process of thousands of tokens. By ordinary intuition, to improve reasoning ability, one must apply training signals to a large number of tokens along this trajectory and let dense updates gradually make the model stronger. This study starts precisely by questioning this premise: what happens if these training signals are deleted one by one?

The study focuses on the scenario of online policy distillation (OPD). Its key difference from traditional knowledge distillation is that the reasoning trajectory is generated by the student model itself, and the teacher model then provides feedback on each step the student actually takes. Standard OPD compares the teacher's and student's probabilities for the current token one by one along the entire trajectory: if the teacher thinks the token should be more likely, its probability is increased; otherwise, it is decreased. A single trajectory can therefore produce thousands of training signals, and this dense token-level signal is usually regarded as an important advantage of OPD.

The research team first conducted an extreme experiment: for each reasoning trajectory generated by the student model, only one token was randomly selected to participate in loss function computation and backpropagation. The model still generated the complete reasoning process for a problem, but only one random position actually produced a training signal. If a response has 2000 tokens, the actual training participation ratio is only about 0.05%, meaning about 99.95% of the training signal was deleted.

By ordinary intuition, such a sparse signal should make training basically ineffective. But the result was the opposite: randomly supervising one token still reliably improved the student model's mathematical reasoning ability. Moreover, this was not an accidental phenomenon on some particular teacher or student—the research team constructed 9 different teacher-student configurations on the Qwen3 series, covering larger-parameter teachers to smaller-parameter students, same scale, and smaller-parameter teachers to larger-parameter students, and consistently observed reasoning ability improvements in these settings.

A random single token was already counterintuitive, but the next result was even more unexpected. In OPD, one can measure the degree of disagreement between teacher and student on each token: on some tokens the teacher approves of the current generation more than the student does, while on others the teacher clearly thinks the student should not generate it that way. The research team therefore tried keeping only these most 'extreme' tokens, for example selecting only the one token per trajectory whose probability the teacher most wants the student to increase, or the one token whose probability the teacher most wants the student to decrease.

The results showed that supervising only such 1-2 tokens already matched or even exceeded standard full-token OPD in many experiments. In other words, standard OPD may train thousands of tokens, while extremely sparse OPD trains only one token, yet the latter is not necessarily worse. In a representative Qwen3 experiment, the student model obtained by selecting a single extreme token not only surpassed the standard OPD student, but even surpassed the teacher model providing the supervision. This suggests that what is happening here may not be as simple as 'fully copying the teacher to the student.'

The mathematical objective of OPD is essentially to reduce the distributional difference between teacher and student. A very natural explanation is that sparse OPD works perhaps only because a small number of tokens is already enough to make the student closer to the teacher. But the experiments do not fully support this explanation.

The research team found that some sparse OPD models with the best reasoning performance did not have smaller reverse KL, and it may even become larger. That is, the student's reasoning ability improved, but it was not necessarily more like the teacher in the overall distribution. From this one can infer (this is an inference based on the above facts) that the reasoning performance improvement brought by sparse OPD does not come only from simple imitation.

There is currently no complete theoretical explanation for why supervision on one token may be useful. The intuition given by the study is that supervision on one token does not mean the model learns only one token; it is more like applying a clearly directed push to the entire model parameters under a specific context. If this push is accurate enough, a small number of key updates can push the model into a better parameter region.

It should be noted that this intuition comes from the research team's explanatory statement and is a source claim rather than a verified mechanistic conclusion. It also points to a more general question: people may have overestimated the amount of token-level optimization that large-model post-training truly needs—this is likewise an inference based on existing evidence, not a proven conclusion.

To answer whether this is only a special phenomenon of OPD, the research team extended the experiments to other environments. The results showed that the similar extreme sparse supervision phenomenon does not exist only in the original combination of Qwen3 plus mathematical reasoning plus OPD, but can also extend to coding reasoning, the Llama series models, and PPO-based RLVR.

Particularly noteworthy is the PPO experiment: even when dense token training cannot significantly improve reasoning performance, extremely sparse token updates may still produce effective improvements. This means the phenomenon may not just be a special property of OPD, but points to a more general question: does post-training really need such dense token-level optimization?

The ablation experiments observed a clear non-monotonic relationship: when the tokens participating in training are gradually reduced, performance may first decline; but when the supervision signal becomes extremely sparse until only a few tokens remain, performance rises again and may exceed the improvement brought by dense signals. This shows that between reasoning performance improvement and supervision signal density, it is not simply 'the more, the better.'

But boundaries must be clearly drawn. This work is not saying that large numbers of tokens are useless, nor is it claiming that future large-model training will need only one token. What is truly noteworthy is the non-monotonic relationship between signal density and performance, and the question it raises: how many tokens in a trajectory are actually driving learning?

If only a very small number of updates in a reasoning trajectory of thousands of tokens is enough to push the model to become stronger, this may inspire more token-efficient and memory-efficient training methods—this is an inference based on existing results, not a verified engineering conclusion. In addition, this conclusion comes from a report by the Amazon and Duke University team submitted to QbitAI, with no third-party independent reproduction provided; the core experimental environment is Qwen3 plus mathematical reasoning plus OPD, and other environments are extended verification; the report did not disclose specific benchmark scores, model scales, or training compute details, nor did it give peer review status. Why extreme sparse supervision works, the relationship between reverse KL not becoming smaller or even becoming larger and reasoning improvement, the transferability of this phenomenon to larger-scale models and real production training, and the stability and reproducibility of the single extreme token selection strategy all remain unresolved questions.