Tencent Hunyuan ACL 2026 Paper: 15.3% of SFT Samples Are 'Fake Learned' – Loss Convergence Doesn't Mean True Mastery

SFT is the mainstream method for adapting LLMs in the industry. The default practice is to prepare labeled data, run SFT training, and consider training complete when the loss curve converges. However, the Tencent Hunyuan team conducted a comprehensive investigation across multiple model families including Qwen, LLaMA, OLMo2 (1.8B-14B) and 10 datasets, finding that on every model and every dataset, there is a batch of training samples—the model has seen them during training, loss has decreased, but when retested later, the answers are wrong. The average ratio is as high as 15.3%. The paper defines this as the 'Incomplete Learning Phenomenon' (ILP) and points out that it is a systematic blind spot in SFT training.

For detection, the paper converts each SFT data point (QA pair) into a 4-choice multiple-choice question using GPT-4, and applies a four-level quality filter. Then, using the pass@k metric (k=5), each MC question is sampled 5 times; if all 5 samples are wrong, it is judged as an ILP sample. Triple verification (cross-temperature consistency, retest reliability Kappa=0.89, cross-model consistency) ensures the reliability of the judgment.

The paper proposes a 2×2 attribution matrix, with the vertical axis being 'whether the base model already possesses the knowledge' (pre-training stage) and the horizontal axis being 'whether the SFT label is correct' (annotation quality). Through a two-step rapid localization: first, zero-shot testing on the base model to determine the knowledge direction; then, comparing early-20% and late-20% checkpoints to determine the training direction, ultimately locking down five major root causes.

One key finding is that ILP is independent of annotation quality: even when SFT label accuracy exceeds 98%, ILP still exists, indicating that this is not a 'wrong data' problem but a systematic deficiency in the SFT training mechanism itself. Another surprising finding is that ILP is weakly correlated with model scale: from Qwen-1.8B to Qwen-14B, ILP only decreases by 2.1 percentage points, suggesting that larger models cannot significantly solve the 'can't learn' problem.

The paper designs five types of interventions corresponding to the five root causes. The most striking comparison experiment is: CPT (2x training amount) reduces ILP by 12.5%, while adding epochs (10x training amount) only reduces it by 2%. This means that adding SFT training epochs is almost useless; the real bottleneck lies in the knowledge reserve during pre-training. SFT itself has a physical upper limit: it can only 'reorganize' existing knowledge, not create new knowledge out of thin air.

Physical trace verification further supports this conclusion: by retrieving the knowledge points involved in ILP samples from the Dolma 5T corpus, it was found that 19.3% of knowledge points are completely absent (Root Cause I: Knowledge Missing), and 14.5% have conflicting information (Root Cause II: Knowledge Conflict).

The paper's findings have important practical implications. First, don't blindly trust loss convergence: training completion does not mean learning is complete; sample-level diagnosis using pass@k + MC conversion is needed. Second, adding epochs has extremely low cost-effectiveness: 10x SFT compute for a 2% improvement; it's better to invest the budget in pre-training data. Third, knowledge missing is the top root cause; if the downstream task involves a lot of new knowledge the model hasn't seen, SFT is almost powerless, and CPT is needed first. Additionally, data deduplication and consistency are low-cost, high-benefit measures; global shuffle and dynamic bucketing are simple to implement, and ROUGE-L can improve by 29%. Finally, the 2×2 matrix can locate root causes within minutes without full retraining.