ERNIE 5.5 Flash Base 280B: Baidu sparse MoE model

What changed

Baidu has released ERNIE 5.5 Flash Base, a 280-billion parameter sparse mixture-of-experts model designed as a foundation for teams building agents and automation workflows. The model activates 14 billion parameters per token, trading total model size for computational efficiency. This is a raw pre-training checkpoint, meaning no instruction tuning or reinforcement learning has been applied, giving developers a clean starting point for their own customization. The weights are available in FP8 and BF16 precision on Hugging Face and ModelScope.

The architecture uses 43 layers with a hybrid attention scheme. Thirty-seven layers employ Heavily Compressed Attention that caches one entry per 128 tokens, while 6 layers retain full-context access through Multi-head Latent Attention with sparse attention. Each mixture-of-experts layer includes 512 routed experts with top-10 selection plus one shared expert. The model was pre-trained on 33.5 trillion tokens with a maximum training context of 128,000 tokens and supports up to 1 million tokens during fine-tuning and inference.

The attention mechanisms include several efficiency features. Low-rank gated attention reduces gate-projection parameters by 75 percent while maintaining performance. Virtual Head Attention adds a lightweight low-rank mix across heads. The model uses multiple independent routers instead of a single router across mixture-of-experts layers, with each router initialized separately to develop distinct expert preferences.

Deployment support includes SGLang and vLLM version 0.32.0 or later with the vLLM-ERNIE plugin. Reference configurations use 4-way tensor parallelism. Baidu trained the model on SM90 and SM100 GPUs using a custom kernel called FlashMask-4, which delivers up to 175 percent faster forward passes and 74 percent faster backward passes compared to FlashAttention-4 MaskMod.

Why it matters

The efficiency profile makes ERNIE 5.5 Flash Base relevant for teams constrained by compute budgets. With only 14 billion of 280 billion parameters active per token, the model reduces memory and arithmetic requirements compared to fully dense alternatives. This matters for building agents and automation workflows where latency and throughput compound operational costs across many inference calls.

The release as a base checkpoint, not a finished chat model, targets researchers and practitioners who want to fine-tune or apply reinforcement learning from a shared foundation. This approach removes uncertainty about what instruction-tuning or reinforcement learning decisions Baidu made, letting teams reproduce or diverge from those choices deliberately.

Baidu’s benchmark results position the model competitively across multiple domains. The company reports MMLU-Pro scores of 78.4, exceeding DeepSeek-V4-Flash-Base at 69.2. On SimpleQA it scores 34.1 versus 29.1 for DeepSeek. On code tasks, it achieves 82.3 on HumanEval+ and 81.5 on MBPP+. It scores 91.7 on RepoQA, suggesting strong long-context reasoning over code repositories. Across math and reasoning, it achieves 87.0 on MGSM and 77.1 on BBH direct.

The model does not lead on all benchmarks. DeepSeek-V4-Flash-Base outperforms it on ChineseSimpleQA, INCLUDE-lite, HELMET-Rerank and InfiniteBench. Tencent’s Hy3-preview-Base leads on BBH chain-of-thought and FullStackBench. These gaps demonstrate that different models retain different strengths depending on the evaluation setup and domain focus.

The open release is available under a license that permits commercial use up to 50 million dollars in annual revenue but excludes deployment in the European Union, United Kingdom and South Korea. This geographic restriction shapes which teams and organizations can use the model freely.

What to test

Before adopting ERNIE 5.5 Flash Base for production agent workflows, validate these vendor claims:

Benchmark reproducibility. Baidu’s numbers come from a unified evaluation pipeline. DeepSeek reports MMLU-Pro at 68.3; Baidu measures 69.2 on the same model in its pipeline. Tencent reports Hy3 MMLU-Pro at 65.76; Baidu measures 66.9. Run your own inference and benchmarking on the released weights to confirm performance on tasks relevant to your agents or automation workflows.

Context window behavior. The model supports 1 million tokens during inference but was pre-trained with a 128,000-token context window. Test whether long-context performance degrades gracefully beyond training context or shows cliff effects. Verify latency and memory usage at maximum context lengths, especially in production deployment where batch sizes and parallel requests matter.

Mixture-of-experts routing. The model uses multiple independent routers instead of a single router. Test whether expert load balancing remains even across different input distributions. Unbalanced routing can degrade throughput and increase per-token latency, offsetting the efficiency gains of the sparse architecture.

Fine-tuning efficiency. Baidu positions this as a foundation for instruction tuning and reinforcement learning. Measure the actual compute and memory required to fine-tune on your domain, and compare the cost against dense base models of similar training capacity. Verify that the sparse architecture reduces fine-tuning cost as well as inference cost.

Deployment overhead. The model requires SGLang or vLLM version 0.32.0 or later with the vLLM-ERNIE plugin. Test setup and deployment time in your infrastructure, and confirm GPU memory usage against Baidu’s reference configurations. Measure inference latency and throughput under production-like batch sizes.

The conclusion

ERNIE 5.5 Flash Base presents a practical option for teams building agents or automation systems where inference cost matters. The 14-billion active parameter count and hybrid attention design reduce memory footprint and latency compared to dense 280-billion-parameter models, though not to the degree of much smaller decision models. Baidu’s benchmark results are competitive but not universal; the model excels on reasoning and code tasks but trails on others.

The release as a raw pre-training checkpoint aligns with how teams actually use base models: as starting material for domain adaptation and fine-tuning. The geographic licensing restrictions and the absence of chat or instruction-tuned variants mean teams must perform post-training work themselves, which raises the barrier for users seeking a ready-to-deploy agent model. The requirement for recent versions of SGLang or vLLM adds infrastructure dependencies that may not align with all deployment environments.

Community fine-tuned versions and instruction-tuned derivatives will likely emerge on Hugging Face. Real-world comparisons of fine-tuning cost and agent performance against similar MoE alternatives will clarify whether the architectural gains translate to production value. GPU memory and throughput measurements from practitioners deploying the model in SGLang and vLLM will reveal whether the theoretical efficiency improvements hold at scale and whether the multi-router design maintains balanced expert utilization in practice.

Related reading

• Prime Mega MoE speeds up mixture-of-experts training 34%
• Gemini Omni 1.2 Flash: 30% price cut, 15-second video
• StateBench: Language model state tracking benchmark