Mixture of Experts (MoE) is a neural network architecture pattern in which a layer is replaced by a set of expert sub-networks plus a router (or gating function) that selects which experts to activate for each input token. Only the selected experts contribute to the forward pass, so the number of parameters touched per token is far smaller than the total parameter count. This decouples model capacity (total parameters) from compute (parameters per token), allowing models with hundreds of billions of total parameters to run at the inference cost of much smaller dense models. MoE became the dominant design for large open-weight LLMs from 2024 onward: early examples include Mixtral 8x7B / 8x22B (Jiang et al., 2024), DeepSeek-V2 / V3 / R1 (DeepSeek-AI, 2024 / 2025), and Qwen2-MoE, and GPT-4 and Gemini 1.5 were reported to use it. By September 2026 most frontier open-weight releases are MoE models, for example DeepSeek-V4.1-Flash (552B backbone parameters, 1 shared plus 384 routed experts per MoE layer with 6 routed experts active per token) and Xiaomi’s MiMo-V2.6-Pro (1.02T total, 42B active). See the LLM landscape for current models.

Mechanism

A standard transformer block has a self-attention sublayer followed by a feed-forward (MLP) sublayer. In an MoE transformer, the MLP sublayer is replaced by N experts (each itself an MLP) and a router. For each token:

  1. The router (typically a small linear layer followed by a softmax) computes a score for each expert given the token’s hidden state.
  2. The top-k experts are selected by score. Early MoE LLMs such as Mixtral used a few large experts with k=2; newer “fine-grained” designs in the DeepSeek line use hundreds of small routed experts, activate several per token, and add always-on shared experts.
  3. Each selected expert processes the token; outputs are weighted by the router scores and summed.
  4. Tokens not routed to a given expert do not consume its compute.

Total parameter count = (parameters per expert) × N + shared parameters. Active parameters per token ≈ (parameters per expert) × k + shared parameters. For Mixtral 8x7B: 8 experts × ~7B per expert (in MLPs) → ~47B total parameters but only ~13B active per token (Jiang et al., 2024).

The router is typically trained end-to-end with the rest of the model. To prevent expert collapse (the router routes everything to a few experts), training adds an auxiliary load-balancing loss (Shazeer et al., 2017; Fedus et al., 2022) or uses expert-choice routing (Zhou et al., 2022) where experts pick tokens rather than tokens picking experts.

Why MoE Matters

The empirical result, established across Switch Transformer (Fedus et al., 2022), GLaM (Du et al., 2022), and the Mixtral / DeepSeek line: at fixed compute budget, MoE models reach lower loss than dense models with the same active-parameter count, because the additional capacity in the un-activated experts encodes specialised knowledge the router can dispatch to as needed. The trade-off is memory, all experts must be loaded for routing to be possible, even though only k are used per token.

For inference economics, this means MoE models offer dense-model-like speed with much-larger-model-like quality, at the cost of dense-model-like-N× memory. This shifts the bottleneck from compute to memory bandwidth, with implications for serving infrastructure (high-memory GPUs, model parallelism strategies that shard experts across devices).

Routing Variants

  • Token-choice top-k routing (original MoE; Shazeer et al., 2017). Each token picks its top-k experts.
  • Expert-choice routing (Zhou et al., 2022). Each expert picks its top-N tokens. Naturally load-balanced; may drop some tokens.
  • Hash routing (Roller et al., 2021). Tokens routed by a fixed hash; simpler but less learned specialisation.
  • Soft mixing / SoftMoE (Puigcerver et al., 2024). All experts process all tokens, weighted by router scores; differentiable end-to-end but loses sparsity savings.
  • Sparse upcycling (Komatsuzaki et al., 2023). Initialise an MoE from a dense model checkpoint; cheaper than training MoE from scratch.

Engineering Considerations

  • Memory dominates. The full expert set must be resident for routing. For Mixtral 8x7B, this is ~47B parameters in memory but ~13B in compute per token. Plan VRAM accordingly.
  • Expert parallelism. Shard experts across devices so each GPU holds a subset; route tokens across devices via all-to-all collectives. The communication cost is the dominant inference overhead in distributed MoE serving.
  • Batching dynamics. MoE throughput depends on token-to-expert distribution. If a batch is dominated by tokens routing to the same few experts, the others are idle. Production servers (vLLM, TensorRT-LLM, SGLang) handle this with expert-aware scheduling.
  • Quantisation interacts with sparsity. Quantising an MoE is harder than quantising a dense model because outliers concentrate in specific experts; specialised methods (e.g. QMoE; Frantar & Alistarh, 2024) are required to preserve quality at low bit-widths.
  • Fine-tuning is sensitive. Naive fine-tuning can destabilise the router. LoRA on the experts and frozen router is a common, stable approach.

When MoE Wins (and When It Does Not)

MoE is the right choice when:

  • Inference compute (tokens/sec/$) is the bottleneck and memory is available.
  • The training set is large enough to fill the additional expert capacity (small datasets do not benefit; experts collapse to redundancy).
  • Latency targets allow the additional all-to-all communication of distributed serving.

Dense models remain preferable when:

  • Memory is the bottleneck (single-GPU deployment, edge inference). A dense 13B model fits where Mixtral 8x7B does not.
  • Cold-start latency matters and loading 47B of weights is too slow.
  • The task is narrow and the additional capacity has nothing to specialise on.

Sources and Further Reading