Prime Mega MoE speeds up mixture-of-experts training 34%

Prime Intellect has released Prime Mega MoE, a GPU kernel that accelerates training for mixture-of-experts models by up to 34%. The company built on DeepSeek’s open-source Mega MoE kernel by adding a backward pass, which DeepSeek’s original implementation lacked. DeepSeek’s kernel only performed forward passes, making it suitable for inference but not for training workflows. Prime Intellect’s fused, overlapping backward implementation enables full training capability for teams building large MoE architectures.

What changed

Prime Intellect announced Prime Mega MoE as an extension within DeepSeek’s DeepGEMM library. Full training steps on Qwen3 MoE models run 32% to 34% faster than the prime-rl baseline trainer. Against TorchTitan, the gains reach 48% to 55% faster training steps. The kernel supports BF16 precision on both NVIDIA Hopper and Blackwell GPUs, with blockwise FP8 support on Hopper and MXFP8 support on Blackwell. Prime Intellect tested the kernel on two Qwen3 models: Qwen3-30B-A3B and Qwen3-235B-A22B, both configured with 128 experts and top-8 routing. Tests ran on 8 x H200 GPU nodes, using 32 to 128 total GPUs per configuration. All reported numbers reflect FP8 precision, 16,384-token sequences, and full activation checkpointing, with measurements taken from training steps 6 to 20.

Why it matters

MoE model training is fundamentally communication-bound. Each token routes to a subset of experts distributed across different GPUs. Those tokens must travel between GPUs during both the forward and backward passes. Standard implementations treat these operations sequentially, forcing the GPU to dispatch tokens, wait for transport, compute, and wait again on the return trip. These idle periods waste GPU capacity.

Prime Mega MoE fuses multiple operations into a single kernel call. While one token group is in transit between GPUs, the kernel computes on a different group that has already arrived. The implementation also skips intermediate tensors by using GPU-only scratch buffers, reducing memory overhead. For teams training large MoE models, the gains translate directly to fewer GPU hours and lower training costs. The 34% reduction in training time per step compounds across hours or days of training runs.

The kernel integrates as a dispatch backend in Prime Intellect’s prime-rl trainer. Developers set type = “mega_moe” under the trainer configuration and choose their preferred precision mode. The training kernels come in a prebuilt DeepGEMM wheel from prime-kernels. The standard DeepGEMM library does not include them, so teams must use Prime Intellect’s distribution to access this feature.

The performance gains vary significantly between forward and backward passes. On the smaller Qwen3-30B-A3B model, the forward pass runs 3.11x faster. On the larger Qwen3-235B-A22B model, the backward pass runs only 1.24x faster. This asymmetry matters because actual training speedups depend on where the bottleneck sits in your specific workflow. Batch size, sequence length, and checkpoint strategy all affect whether forward or backward operations dominate the training loop.

What to test

Before adopting Prime Mega MoE for production training, verify these claims independently:

• Reproduction on your hardware: All reported numbers come from tests on 8 x H200 nodes with specific configurations (FP8, 16,384-token sequences, full activation checkpointing). Performance on different GPU counts, types, or with partial checkpointing may differ substantially from published results.

• Backward pass gains vs. forward gains: The forward pass shows 3.11x speedup on the smaller model, but backward pass gains are much smaller (1.24x on the larger model). Measure the actual bottleneck in your workflow to see where the wins apply. If backward passes dominate your training, the overall speedup may be significantly lower than the headline 34%.

• Precision mode trade-offs: Prime Intellect offers BF16, blockwise FP8 on Hopper, MXFP8 on Blackwell, and a hybrid fp8_bwd mode (BF16 forward, FP8 backward). Test whether lower-precision modes introduce convergence or accuracy drift in your models or whether they provide sufficient numerical stability for your use case.

• TorchTitan baseline setup: The 48% to 55% gain over TorchTitan comes from a different configuration (B200 GPUs, 671B model, BF16 baseline). Ensure the comparison reflects your target setup before drawing conclusions about relative performance.

• Integration with prime-rl: The kernel only works through prime-rl’s mega_moe dispatch backend. If you use a different trainer or framework, you cannot use this kernel. Confirm that prime-rl fits your existing training pipeline and dependencies.

• Architecture coverage: These results apply specifically to Qwen3 MoE models with 128 experts and top-8 routing. Other MoE architectures or routing strategies may exhibit different performance characteristics.

The conclusion

Prime Mega MoE addresses a real bottleneck in mixture-of-experts training by fusing communication and computation. The 32% to 34% speedup on full training steps is substantial and appears consistent across two model sizes. The engineering choice to overlap token transport with computation on different groups is sound, and the addition of backward pass support fills a gap in DeepSeek’s original work.

The key limitation is scope. This is a kernel for a specific hardware and framework combination. It works on NVIDIA Hopper and Blackwell GPUs through prime-rl, and only on models with the Qwen3 MoE architecture tested. Forward pass gains are much larger than backward gains, so actual training speedups depend on your batch size, sequence length, and checkpoint strategy. Teams training other MoE variants or using different frameworks will not benefit.

Watch for two developments: whether Prime Intellect ports the kernel to other MoE architectures beyond Qwen3, and how the backward pass performance evolves as the implementation matures. A 1.24x backward gain on large models is useful but not exceptional. Incremental gains here would move the needle further on total training time.

For teams already committed to Qwen3-based MoE training on NVIDIA GPUs, Prime Mega MoE’s cost savings are worth testing in a sandbox environment. For others, verify that the architecture and framework fit before integrating it into your pipeline.

Related reading

• GPT-6.1 Sol vs Claude Opus 5.5: Cost and Performance
• GPT-6.1 Terra vs Claude Opus 5.5: Pricing and Performance
• Synthesia vs Colossyan: AI Training Video