Baidu has released and open-sourced ERNIE 5.5 Flash Base, a sparse mixture-of-experts model with 280 billion total parameters that activates 14 billion per token.
The model was pretrained from scratch on 33.5 trillion tokens with a maximum training context of 128,000 tokens. Fine-tuning and inference support a context window of one million tokens.
What was released, and what was not
The checkpoint published is the endpoint of pretraining. No mid-training and no post-training have been applied. Baidu frames this as giving developers a single common starting point from which they can run domain adaptation, instruction tuning, reinforcement learning for reasoning and agent training.
Weights are published on Hugging Face in FP8 and BF16. Baidu has also published a vLLM plugin and an SGLang cookbook entry so the model runs on the two most common serving stacks.
On Baidu’s published evaluations
Baidu compares the model against base models of comparable size: DeepSeek-V4-Flash-Base at 284 billion parameters with 13 billion active, Tencent’s Hy3-preview-Base at 295 billion with 21 billion active, and Xiaomi’s MiMo-V2.5-Base at 310 billion with 15 billion active. The company reports leading results across several capability dimensions and claims best-in-class status at this size, with the widest margins in reasoning, code, academic subjects and long-context tasks.
The efficiency work sits in three places
The architecture is 43 Transformer layers with a hidden size of 4,096 and 64 attention heads. Three changes carry most of the efficiency claim.
The first is a hybrid sparse attention scheme that combines heavily compressed attention with multi-head latent attention, so different layers handle historical information differently. Thirty-seven layers compress the key-value cache hard, and six layers retain precise retrieval. Low-rank gated attention and a virtual-head attention mechanism sit inside the attention block to improve gating and the fusion of information across heads.
The second is a multi-router latent mixture-of-experts layer, which changes how the model selects experts. The third is a future-token prediction mechanism called ECHO, which lets the backbone learn about the next tokens ahead of generating them. A MultiMax output head and an R2L tokeniser complete the model-side changes.
On the training side Baidu uses the Muon optimiser with FP8 mixed precision, a distributed implementation of Muon, and a new attention operator called FlashMask-4 built for large head dimensions, aimed at communication, memory use and compute efficiency at scale.
The data mix
Three data families were emphasised. Web text came from large-scale multilingual crawling with high-fidelity extraction and model-driven quality filtering. Code combined static repositories and pages with commits, issues, pull requests, code review, requirement-to-PR pairs, multi-commit trajectories, automatically generated tests and CI-verified fixes. Long documents combined naturally long text with quality filtering, concatenations of real text and semantically similar distractors, and synthetic retrieval, citation-grounded question answering and multi-hop question answering data.
Repository-level code and long-range dependency handling are the two capabilities that are hardest to bolt on after pretraining, which is why they were pushed into this stage.
Editor’s note: This is an adapted translation of the original Zhidx report. It has been trimmed and restructured for readability for an international business audience. The full original (in Chinese) is at https://www.zhidx.com/p/607749.html.