Xiaomi's HySparse2 targets lower computing demands for long-context AI agents

THE ESSENTIALS

At one million input tokens, the paper reports about one-fifth of Hybrid SWA's prefill computation and a smaller KV cache.
The comparison uses different attention configurations, so the memory gains cannot be attributed solely to cross-layer sharing.
The results concern research models and specific workload measures; they do not establish a fivefold improvement in product speed.

Less work before an agent answers

Xiaomi's LLM-Core team has detailed HySparse2, an attention architecture aimed at AI agents that repeatedly read tool results and long conversation histories. The proposal targets the work a model does to process those inputs before generating its next response.

The design shares cached information across layers and lets input processing stop after the first part of the model. It also selects individual relevant tokens from a long context, rather than retaining whole blocks around them.

In the paper's analysis of an 80B-A3B research configuration at one million tokens, HySparse2 needs about one-fifth of Hybrid SWA's prefill computation and roughly one-third of HySparse's. With FP8 cache storage, its KV cache occupies 2.69 GB, compared with 12.09 GB and 6.72 GB for those respective baselines. This cache holds information from earlier tokens for reuse during generation.

How to interpret the results

The memory comparison includes more than a change in cross-layer sharing. HySparse2 uses multi-query attention, while both baselines use grouped-query attention. The paper identifies that configuration difference as another contributor to its smaller cache.

The authors also report better long-context retrieval after training the comparison models on the same data and schedules. At 256,000 tokens, HySparse2 scores 58.45 on RULER-v2, versus 32.61 for HySparse and 35.74 for Hybrid SWA.

These findings need to be kept separate: the million-token figures describe computation and cache storage, while the cited retrieval result is at 256,000 tokens. Neither figure establishes an equivalent improvement in an application's response time or operating cost.

For developers, the paper presents a possible way to reduce the overhead of agents reading growing histories. The reported ratios apply to the authors' research configurations; deployment decisions would still require measurements on the intended hardware and workload.