Shanghai AI Lab and SJTU Propose NCP Pretraining Task, Open-Source First 8.9B Latent Space LLM

Shanghai AI Laboratory and Shanghai Jiao Tong University's LUMIA Lab jointly released a technical report introducing Next Concept Prediction (NCP), a new pretraining task, and open-sourced NCP-ArchPreview, the first 8.9B discrete latent space foundation model. Trained on the 5.73T Dolma-3 corpus, the model matches OLMo-3-7B's final pretraining loss using only 51.3% of the token budget, achieving a 1.95x effective convergence speedup and nearly 6 points improvement on GSM8K. All training checkpoints, weights, and evaluation code have been open-sourced, potentially signaling a shift from token prediction to concept prediction in LLM pretraining.