Xiaomi discloses MiMo-V3 plans to adopt HySparse2 architecture, stating that prefill computation for million tokens can be reduced by approximately 80%

Article

Xiaomi first disclosed MiMo-V3 a new architecture: prefill compute for million-token inputs reduced 80%Beating AI news reported that Xiaomi's MiMo team disclosed the next-generation MiMo-V3 will adopt the HySparse2 architecture. It mainly addresses the problem of Agent becoming increasingly expensive.Agent sends only very short instructions each time, but may retrieve large amounts of content from webpages, terminals, and tools. The model must first read all this new content; the longer the context, the more compute this step consumes,KV Cache and the more GPU memory it occupies.HySparse2 first splits the model into a front half and a back half. The front half processes the input, while the back half continues inference and generation. The context information needed by the back half can be generated directly from the results already computed by the front half, so prefill does not need to run the back half in full again. This idea comes from Microsoft's Research Institute's 2024 proposal of YOCO; its name is “You Only Cache Once.” The core idea is to let the back half share information already computed by the front half, storing less cache and performing less duplicate computation.

It also redesigns sparse attention. Ordinary full attention must examine the entire context each time, while HySparse2 allows only a small number of layers to do so. These layers first select the most relevant 1024 items token from the long context, and subsequent layers directly reuse these results while always retaining the most recent 128 items token. The previous-generation HySparse grouped every 64 items token into a group before selecting; as long as one group contained important information, the entire group had to be retained. It now selects token individually. With the same amount of computation, more genuinely relevant content can be retained. In separate tests of this change, the scores on two long-text retrieval benchmarks increased by 6.57 points and 8.14 points, respectively.

The paper uses a model with a total of 80000000000 parameters and approximately 3000000000 activated parameters per step for comparison. The 49-layer model needs to run only the first prefill layers during the 25 stage. When the input reaches 1000000 token, compared with the hybrid sliding-window attention used by the MiMo-V2 series,prefill compute is reduced to approximately 1/5，KV Cache from 12. 09GB to 2. 69GB.

Original link https://m.theblockbeats.info/flash/368764

Key points

The Xiaomi MiMo team disclosed the next-generation MiMo-V3 architecture to be adopted by HySparse2.
HySparse2 The model is divided into front and rear sections, with the rear section able to use results already computed by the front section.
HySparse2’s sparse attention selects the most relevant 1024 items token and always retains the most recent 128 items token。
After separately testing the change to select token individually, the scores on two long-text retrieval benchmarks improved by 6.57 points and 8.14 points, respectively.
of the total parameters 80000000000, with approximately activated each time 3000000000 of the parameters 49-layer model, million token when inputting prefill only need to run the first 25 layers.
million token when inputting,prefill reduces computation to approximately 1/5，KV Cache from 12. 09GB to 2. 69GB。
The article states that this approach originated from Microsoft Research 2024 proposed in YOCO。

AI-assisted interpretation

The following is analysis, separate from reported facts. Verify important claims independently.

Agent An Agent may issue only a short instruction but obtain a large amount of content from webpages, terminals, and tools. The model must first read this content, which increases computational load and GPU-memory usage.HySparse2 allows the model to reuse information from the first half in the second half while focusing primarily on only part of the token, so the article's tests show that the computational load and KV Cache usage for long inputs decrease.

Why it matters to readers

If the test results reported in the article can be reproduced and incorporated into MiMo-V3, the computational and GPU-memory Agent pressure of long contexts may be reduced.

This indicates that when processing very long inputs, the model does not necessarily need every layer to read the entire content; architectural design can reduce redundant computation and cache usage.

Risks and unknowns

The evidence is a media news brief paraphrase and does not provide the original Xiaomi paper or official announcement.
Not confirmed by the evidence HySparse2 whether it has already entered formal MiMo-V3 models or external services.
The evidence does not provide complete testing methods; retrieval performance and resource metrics still require independent verification.

Xiaomi discloses MiMo-V3 plans to adopt HySparse2 architecture, stating that prefill computation for million tokens can be reduced by approximately 80%

Article

AI-assisted interpretation

Why it matters to readers

Risks and unknowns

Related Developments

Related concepts

Keep following

Security Center