How Mixture of Experts Scales Model Parameters
Mixture of Experts lets a model’s parameter count grow tens of times over while the compute per token barely moves. We build it up step by step: copying the FFN into experts, the router, dispatch and combine, top-K routing, and the auxiliary loss that keeps experts balanced.