Luo Fuli Teases Xiaomi MiMo-V3 Architecture: HySparse2 Enables Early Prefill Exit

MiMo-V2.6 had just launched, and Luo Fuli began teasing MiMo-V3. According to QbitAI, MiMo-V3 will adopt a new architecture, with HySparse2 at its core, aimed at the increasingly heavy input-processing problem when Agents execute long tasks. An Agent may need only a few dozen tokens to generate one tool call, but search results, web pages, code, and execution logs can come back in tens of thousands of characters at once; as rounds increase, the model must both process new input and retain enough history for later retrieval.

This creates a clear trade-off: the team wants the model to spend less effort processing input and occupy less memory storing history, while still accurately finding clues in very long records. It must be emphasized that MiMo-V3 has not been released; what is currently public is design and analysis at the architecture-paper level, and latency and task performance in the actual product must wait until the model lands.

Xiaomi's earlier HySparse already used sparse attention, letting a small number of full-attention layers view more complete history, with subsequent sparse layers reusing the content and cache they selected. But long input still had to pass through all model layers during the prefill stage. HySparse2 further adjusts inter-layer dependencies so prefill can end midway: the model is divided into a front self-decoder and a rear cross-decoder, with the front combining full attention and sliding-window attention, and the rear combining full attention and sparse attention.

KV Bridging between the two sections connects only the full-attention layers: the rear full-attention layers generate K and V from the hidden states of the corresponding front layers, while retaining their own independent projection parameters; KV Reuse within the rear section lets sparse layers reuse the KV Cache and token selection results of the full-attention layers in the same hybrid module. The two-level KV sharing described in the paper refers to these two designs.

Cross-layer sharing alone is not enough. In the previous generation's design, sparse layers had an additional sliding-window attention branch, whose cache depended on hidden states computed layer by layer in the rear section; if retained, long input would still need to enter the rear section to build cache, and prefill could not fully exit early. HySparse2 removes this independent branch and instead forcibly includes the most recent tokens in sparse selection, so local information is still retained but uses the same shared KV Cache as distant selected tokens. The model presented in the paper has 49 layers in total; when prefill and generation are deployed separately, the prefill node only needs to hold the first 25 layers and related projection modules, requiring nearly half the model memory; in this configuration, only one layer performs full attention during the prefill stage.

After reducing input computation, whether information in long history can be accurately retrieved depends on how sparse layers select content. The previous generation HySparse selected content by blocks; even if only a small amount of content in a 64-token block was relevant, the entire block would consume selection budget. The paper points out that this approach showed no obvious disadvantage in early pretraining evaluations, but in Agent tasks spanning multiple rounds of tool calls, the problem of insufficient retrieval precision became prominent.

So HySparse2 changes selection granularity to individual tokens. The paper's configuration selects 1024 global tokens and then forcibly retains the most recent 128 tokens, so it can both search earlier history for scattered clues and not miss the tool results just received; subsequent sparse layers continue to reuse these selected positions and their cache. For an Agent that needs to find evidence across multiple rounds of tool calls, who gets the limited attention slots directly affects whether it can connect earlier clues to the current task.

In ablation experiments, while keeping the backbone structure and attention budget consistent and comparing only block-based versus token-based selection, in long-context tests at 32k and below, token-based selection improved RULER-v2 by 6.57 percentage points, dual-cue MRCR-v2 by 8.14 percentage points, and GraphWalks by 5.55 percentage points. HySparse2 also does not completely remove full attention: the team believes a small number of full-attention layers both help maintain model capability and use complete attention scores to help later sparse layers select tokens worth looking at, eliminating the need to separately train a retrieval module.

With so many architecture changes, ultimately it still comes down to whether the model can do things right. The Xiaomi team compared three attention designs—HySparse2, HySparse, and Hybrid SWA—using the same data and training schedule. In ordinary knowledge, reasoning, and code projects after pretraining, HySparse2's results rose and fell, overall roughly comparable to the control models; its advantage mainly appeared in long-context capability. After adding Agent data and further post-training, the gap became more obvious: HySparse2 led the two control architectures in MRCR-v2 and RULER-v2 retrieval scores across all context lengths evaluated in the paper, and also had lower perplexity related to Agent trajectories and long-distance dependencies.

At 256k context, HySparse2 scored 58.45 on RULER-v2, versus 32.61 for the previous generation HySparse and 35.74 for Hybrid SWA. Averaged by test length, its MRCR-v2 and RULER-v2 scores were 11.30 and 19.81 percentage points higher than HySparse, respectively. On cost, at 1 million tokens, the paper's analysis-derived prefill compute for HySparse2 is about 2.92x lower than HySparse and about 5.02x lower than Hybrid SWA; with FP8 cache, the KV Cache footprints of the three are 2.69GB, 6.72GB, and 12.09GB, respectively.

A boundary must be drawn here: the 5.02x compares prefill compute, not actual response speed; the paper's long-context capability tests evaluate up to 256k, and 1 million tokens corresponds to compute and cache analysis. MiMo-V3 has not been released, and latency and task performance in the actual product must wait until the model lands. Still, HySparse2's orientation is already clear—when an Agent executes tasks, tools continuously send back large amounts of new content, and key clues in history cannot be lost. How much of the paper's improvements MiMo-V3 can bring into actual tasks will be the more noteworthy result next.