Xiaomi Mimo Team Releases MiMo-V3's New Attention Architecture HySparse2

The Xiaomi Mimo team announced HySparse2, a new attention architecture for MiMo-V3, which uses a two-level KV sharing mechanism to simultaneously reduce prefill computation, KV cache size, and improve long-context retrieval accuracy. On an 80B-A3B MoE model at 1M tokens, prefill FLOPs are only 1/5 of the previous Hybrid SWA, KV cache is 1/4.5, while long-context retrieval benchmarks such as RULER-v2 and MRCR-v2 significantly outperform. The HySparse2 paper has been released.

+1