LRMoE: Demand-Side Load Re-Calibration for Efficient MoE Training on Memory-Constrained Nodes

Expert‑mixing (MoE) models expand the capacity of AI systems, but uneven token distribution among experts creates bottlenecks in both computation and communication. Existing approaches try to balance the load by redistributing or replicating experts, which in turn competes for the GPU’s limited memory with the buffers needed for tokens. The new LRMoE system introduces a memory‑aware strategy that simultaneously adjusts demand during back‑propagation and keeps the residual load balanced across each node. In the forward pass, the algorithm sets retention quotas and determines expert placement while respecting resource constraints, without dropping any directed operation. During back‑propagation, a recalibration metric evaluates each token‑expert pair’s contribution to the model update and selects only those that fit within the established quotas. Afterwards, a capacity‑aware routing distributes the retained pairs among the available copies of the original expert, and a physical compaction step creates smaller data packets and variable‑size GEMM operations, reducing communication traffic and buffer pressure. Experiments with the Qwen3‑30B‑A3B and GLM4.5‑106B‑A12B models show that LRMoE shortens the total training time for MoE layers by 21 % to 45 % compared to the SYMI solution. This improvement is significant because it enables training larger, more complex models on hardware with limited resources, facilitating the adoption of advanced AI techniques in environments with constrained infrastructure budgets.

Read the original article on arXiv cs.AI

This summary is an informational synthesis produced by dataqbs.com. All rights to the original content belong to its author and the cited media outlet. We act solely as curators of technology news and claim no authorship.