Qwen releases Flash-Next as an early preview of Qwen4 architecture

An architecture preview with public weights

Alibaba has released Qwen3.8-Flash-Next weights as an early view of design choices intended for Qwen4. The official repository dates the release to August 26. Its later technical paper, submitted on August 31, examines how capability, computational cost and training stability change together.

The main model has 125 billion parameters, with 6 billion activated per token, plus an additional 51 billion parameters in n-gram embedding tables. The extra table is kept outside accelerator memory and prefetched from host memory. Reporting only the main-model figure would omit a substantial part of the design's stored capacity.

Four changes that work together

The repository groups the changes into attention, residual connections, embeddings and optimization. A hybrid of Gated DeltaNet and Qwen Sparse Attention combines compressed history with selective access to context. A gated residual stream has four branches, while the embedding lookup adds capacity with comparatively little extra computation. The optimizer uses a revised Muon-based recipe.

These components address different constraints. Attention controls which earlier information a token can use. Residual connections carry information between layers. An embedding table can add learned associations without activating another large block of computation on every token. The design therefore changes both how information is represented and where the cost is paid.

What the paper's comparison actually measures

The authors compare the new design with a 397B-A17B predecessor on 14 pretraining benchmarks. They report leading on eight and trailing by no more than 2.6 points on the others, with about one third of the activated parameters, one third of the training tokens and roughly one ninth of the training floating-point operations.

That is a defined experimental comparison, not a statement that all end-user tasks cost one ninth as much. Training compute, inference latency and the price of an API call are different quantities. The paper's ablations are useful precisely because they examine several dimensions instead of treating parameter count as the sole measure of efficiency.

Routes to use the release

Qwen identifies three managed or application-level routes: the QwenCloud API, Qwen Code and QwenWork's Standard mode. The repository separately documents local operation with Transformers, llama.cpp and MLX-based software, plus server deployment with SGLang, vLLM and TokenSpeed. The example server configurations use tensor parallelism across four devices and a 262,144-token context setting. That example is an operational configuration, not a statement that every local machine can load the original checkpoint. The model-weight license is linked separately from the runtime instructions.

A useful negative result

The study also reports that increasing the n-gram vocabulary continues to lower training loss while downstream accuracy eventually levels off. This is a substantive qualification: making the model better at its training objective does not indefinitely improve the tasks used to evaluate it.

The authors find that architecture and optimizer choices shift favorable learning-rate and batch-size settings, and improve stability in their stress tests. The release is consequently as much a report about training a model efficiently as it is a new downloadable checkpoint. Qwen positions it as a preview, so it should not be described as the completed Qwen4 family or as evidence that every future Qwen4 variant will use identical settings.

SGLang's deployment guide illustrates the physical consequences of the separate embedding table. Its NVFP4 package is about 126 GiB, including a roughly 47.7 GiB n-gram table. On a single DGX Spark, moving that table to ordinary host memory does not solve the capacity problem because CPU and GPU share the same memory pool. The documented single-machine route instead places the table in a file on NVMe storage.