Mixture of Experts Models: Doing More with Less

Mixture of experts models (MoE) are a core solution to the tension between “parameter expansion” and “computational cost” in large models, and are now being adopted by many leading large models such as GPT, DeepSeek, Mixtral, and others.

What Is Mixture of Experts? Mixture of Experts Explained

Mixture of experts models are a conditional computation architecture that splits the Feed‑Forward Network (FFN) layers of large Transformer models into multiple independent “expert” subnetworks, and introduces a learnable routing (gating) mechanism that activates only a small number of experts for each input token (typically Top‑K, where K=1 means only the single highest‑scoring expert is activated and the rest are completely skipped; K=8 means eight experts are selected). This allows a dramatic expansion of the total parameter count. Mixture of experts models have become one of the core techniques for compute‑efficient scaling of large language models (LLMs), significantly outperforming dense models under the same computational budget.

Simply put, instead of using one massive “generalist” model to handle all tasks, the problem is broken down and handed to a team of multiple “expert” sub‑models, with a “routing” or “gating” mechanism to manage and dispatch them, and their outputs are combined to produce the final answer. This greatly increases the model’s capacity and performance without substantially increasing computational cost.

---

Mixture of Experts Architecture

Expert Layers

The most computationally expensive FFN (feed‑forward network) layers in the Transformer are split into N independent small networks, each of which is an “expert.” Each expert is an independent FFN that excels at handling different types of data or tasks—some are good at mathematics, others at writing. For example, in Mixtral 8x7B, each expert is a small Transformer FFN layer with about 5.6B parameters; the total of 8 experts amounts to roughly 46.7B parameters.

These “specializations” are not manually assigned but emerge automatically during training—a phenomenon known as Expert Self‑Organization.

---

Gating Network / Router

The gating network is the “dispatcher” or “commander‑in‑chief” of MoE—it decides which experts should handle each input, and is one of the most important components of the MoE architecture.

The complete workflow of the gating network:

—Receives the high‑dimensional vector representation of the input token (typically a vector of several thousand dimensions).

—Computes a “matching score” between the token and each expert via a lightweight linear layer.

—Normalizes the scores (Softmax or Sigmoid).

—Selects the top‑k experts with the highest matching scores (the current mainstream is Top‑2, i.e., selecting two experts for better results).

—Sends the input token to the selected experts for processing.

—Weighted‑combines the outputs of each expert according to the gating weights.

—Passes the combined result to the next layer.

Mathematically, the gating network’s operation can be expressed as:

scores=Softmax(W_gate​⋅x_token​)

weights, indices=TopK(scores,k)

output=∑_i∈indices​weights_i​⋅Expert_i​(x_token​)

The gating network itself is usually very lightweight (often just one linear layer plus Softmax/Sigmoid); its parameter count is negligible compared to the expert networks. For example, in DeepSeek‑V3, the gating network has only about a few million parameters, while the expert networks have hundreds of billions—the gating network accounts for less than 0.1% of the total parameters.

---

Sparse Activation

This is the core mechanism of MoE—and the key that distinguishes it from dense models.

Sparse activation means: for each input token, only a small fraction of experts are activated, while the rest remain dormant.

In Mixtral 8x7B: 8 experts, only 2 activated—75% of parameters are dormant.

In DeepSeek‑V3: 256 experts, only 8 activated—about 97% of parameters are dormant.

---

How Mixture of Experts Models Work?

When a token enters an MoE layer, the following steps occur:

—The gating network computes an n‑dimensional score vector (n = number of experts):

raw_scores=W_gate​⋅x_token​

—Converts the raw scores into a probability distribution (Softmax or Sigmoid):

gate_probs=σ(raw_scores)

—Selects the Top‑k experts based on the probability distribution:

selected_experts=argsort(gate_probs)[0:k]

—Performs forward computation only on these k experts (computations for other experts are skipped):

e_i​=Expert_i​(x_token​)for i∈selected_experts

Weighted‑sums the outputs of the k experts by their gating weights:

output=∑_{i∈selected_experts}​gate_probs_i​⋅e_i​
​

—The output is passed to the next layer (typically via residual connection and layer normalization).

Mathematically, the full output of an MoE layer can be expressed as:

y=∑ⁿ_i=1​G(x)_i​⋅E_i​(x)

where:

x is the vector representation of the input token

G(x)_i is the weight assigned by the gating network to the *i*‑th expert (zero for all except Top‑*k*)

E_i(x) is the output of the i‑th expert

n is the total number of experts

---

Typical Mixture of Experts Models

Closed-source Mixture of Experts Models

–Gemini Mixture of Experts (MoE)

Google officially confirmed for the first time that Gemini 3.0 Pro adopts a sparse MoE architecture (previous Gemini versions were never explicitly confirmed by the company to use MoE). It surpasses GPT‑5.1 and Claude Sonnet 4.5 in multiple tests and has been widely rated as “the most capable AI overall.” Its MoE routing mechanism is described as a simplified sparse gating design, similar to Switch Transformer routing but re‑designed for multimodal scenarios.

Unlike most MoE models, Gemini was designed from the outset to handle text, images, audio, video, and code within a single unified architecture. This means its “experts” handle not only different domains (math vs. literature) but also different modalities (image vs. text), making the MoE design far more complex than in pure‑text models.

GPT Mixture of Experts (MoE)

OpenAI has never publicly confirmed the architectural details of GPT, but multiple independent sources and community analyses make MoE the most prevalent speculation:

Parameter estimates: GPT‑4 is rumored to have about 1.8T total parameters, composed of 8–16 experts, each roughly 111B–220B parameters.

Inference cost characteristics: GPT‑4’s inference latency and pricing pattern are highly consistent with MoE models—per‑token cost is far lower than that of a dense model of equivalent scale.

Internal leaks: In‑depth analysis reports from organisations like SemiAnalysis in 2023 pointed out that GPT‑4 uses an MoE architecture to achieve 1.8T parameters while maintaining manageable inference costs.

---

Open‑Source Mixture of Experts (MoE)

Mixtral Mixture of Experts model

This was the first mixture of experts model to have a wide impact in the open‑source community.

Pure and canonical mixture of experts design: 8 experts, Top‑2 routing, no fancy shared experts or fine‑grained segmentation. This is the “fundamentalist” implementation of MoE and serves as the best teaching example for understanding MoE.

Grouped‑Query Attention (GQA): 32 query heads are divided into 8 groups sharing KV heads, significantly reducing memory usage, allowing the 47B model to run (with offloading) on consumer‑grade hardware.

Parameter sharing strategy: Attention parameters are shared across all layers; only the FFNs (experts) are independently trained in the MoE layers. The “bloat” in total parameters comes entirely from increasing the number of expert FFNs.

On mainstream benchmarks like MMLU, HellaSwag, and GSM8K, it matches or surpasses LLaMA‑2‑70B, while being 6× faster in inference. The instruction‑tuned Mixtral‑Instruct achieved a high score of 8.30 on MT‑Bench, making it the strongest open‑source model at the time.

–DeepSeek‑V3 Mixture of Experts (MoE)

Auxiliary‑Loss‑Free Load Balancing
This is one of DeepSeek‑V3’s most original contributions. Traditional MoE must rely on auxiliary loss functions to prevent expert load imbalance, but auxiliary losses can conflict with the primary task loss via gradients, hurting model performance. DeepSeek‑V3’s solution: introduce a dynamically adjustable bias term bᵢ for each expert, and perform Top‑K selection on the routed scores sᵢ + bᵢ. If an expert is overloaded, its bias is reduced; otherwise, it is increased. The bias term does not participate in gradient propagation, so it does not affect primary task training. This strategy maintains load balance throughout training without sacrificing performance due to auxiliary losses.

Multi‑Token Prediction (MTP)
Traditional language models predict only the next token; DeepSeek‑V3 predicts multiple future tokens simultaneously. This not only serves as a data augmentation method (the model receives more supervisory signals from each position) but also can be used for speculative decoding to accelerate inference.

Multi‑Head Latent Attention (MLA)
MLA introduces latent vectors to cache intermediate results during autoregressive inference, reducing FLOPs in generation tasks. It also optimises KV caching by pre‑computing and reusing static key‑value pairs, further improving computational efficiency.

RMSNorm Normalisation
DeepSeekMoE uses RMSNorm instead of traditional LayerNorm, scaling inputs using only root‑mean‑square statistics. This simplified design reduces computation and improves training stability.

FP8 Mixed‑Precision Training
DeepSeek‑V3 is the first to implement FP8 training on a large‑scale MoE model. Through careful quantisation strategies (e.g., higher precision for attention, FP8 for FFNs), it maintains training stability while compressing total training cost to roughly $5.57 million (2.78 million GPU hours)—about 1/10 of that of a dense model of similar scale.

Model	Total Parameters	Active Parameters	Number of Experts	Top‑K	Context Length	Key Innovations
Mixtral 8×7B	46.7B	12.9B	8	2	32K	MoE open‑source benchmark
Mixtral 8×22B	141B	39B	8	2	65K	Medium MoE, good cost‑performance
DeepSeek‑V2	236B	21B	160	6	128K	MLA + fine‑grained experts + shared experts
DeepSeek‑V3	671B	37B	256	8	128K	Auxiliary‑loss‑free balancing + MTP + FP8
Gemini 3.0 Pro	Undisclosed	Undisclosed	Undisclosed	Undisclosed	1M+	Native multimodal MoE, top proprietary model
GPT‑4 (estimated)	~1.8T	~280B	8 to 16	2	8‑32K	First commercial MoE, details confidential

---

Mixture of Experts Models vs. Dense Models

To help readers understand, here is a company‑meeting analogy:

Mixture of Experts Models

A user asks a sales‑related question. The company has hundreds of experts. It first determines that this is a sales issue; it does not call in all experts, but only notifies the most relevant few experts or departments to hold a meeting.

This is analogous to mixture of experts: the model still has a very large total parameter count, but the MoE router, based on the current token, picks only a few experts with the highest scores. The selected experts each complete a small portion of the task, and the router combines their outputs weighted by their scores, passing the result to the next layer. This is how mixture of experts models can continue to increase model capacity while keeping per‑inference cost under control.

Take Kimi K3 as an example: its total parameters are 2.8 trillion, but only 104 billion parameters are activated per token—about 3.7% of the total.

A common misconception: one should not simply label an expert as a “technical expert” or “sales expert.” During training, experts are not manually assigned; they gradually form internal patterns through extensive training. People can only observe from overall performance that they seem to have learned different preferences—like in a company, people gradually notice that Tom is good at programming.

Dense models

A user asks the same sales question; the company convenes all departments—sales, finance, etc.—to meet together. The more departments participate, the more thorough the discussion, but the meeting takes longer and costs more.

Dense models are similar: for every token, the vast majority of core parameters participate in computation. If a model has 1.2 trillion parameters, that typically means nearly 1.2 trillion parameter computations are performed.

As a clear and reliable architecture, dense models have these advantages:

The structure is relatively straightforward, with one less layer of “who to call” decisions during training and debugging.

All capabilities work together within the same network, with no worry that the router might send the problem to the wrong expert.

Their disadvantage: the larger the model, the more computation is required per generated token. At the trillion‑scale, inference becomes expensive.

Table: Mixture of Experts Models vs. Dense Models

Comparison Dimension	MoE (Mixture of Experts) Models	Dense Models
Total parameter count	Trillion‑scale (large)	Up to tens of billions (small)
Activated parameters per token	3%–5% of total (small)	100% of parameters (large)
Inference cost for equivalent performance	Low (1/2 to 1/3 of dense models)	High
Memory footprint	Extremely high (must fit all experts)	Moderate
Training difficulty	High (load balancing, routing challenges)	Low, mature and stable
High‑concurrency, large‑batch throughput	Extremely high, clear advantage	Average
Small‑batch, low‑concurrency speed	Slow, low cost‑effectiveness	Fast, stable
Multi‑task performance	Good (expert specialisation)	Average
Domain customisation / incremental updates	Convenient (hot‑swap experts)	Complex (requires full fine‑tuning)
Local deployment friendliness	Poor (flagship MoE cannot run on consumer GPUs)	Good; 7B/14B/34B models can all run locally
Best‑suited scenarios	High‑concurrency online APIs, general‑purpose large models for multi‑tasking	Local deployment, low‑traffic internal tools, vertical‑domain small models, on‑device models

---

Advantages of Mixture of Experts Models

–Technical trajectory: Earlier scaling laws for large models were: more parameters → better performance. But this rule breaks down for dense models beyond 70B. From a 70B dense model to a 140B dense model: parameters double, computation doubles, cost doubles, yet MMLU accuracy increases by only 2%—very low cost‑effectiveness. In contrast, for mixture of experts models, total parameters can grow from 70B to 1.6T (20×), while activated parameters increase only from 32B to 49B, computation rises by only 50%, and performance improves by over 15%—highly cost‑effective. In short: dense models hit diminishing returns when scaling parameters further; mixture of experts models offer a technical path to keep scaling up models and improving performance.

–Performance: Mixture of experts models distribute knowledge across different experts—e.g., programming tokens go to programming experts, math tokens to math experts. Each expert focuses on its own domain. With the same number of activated parameters, mixture-of-experts models achieve 5–10% better multi‑task performance than dense models.

–Cost:

Training cost: MoE can use expert parallelism, placing different experts on different GPUs, achieving higher distributed training parallelism and training speeds 2–3× faster than dense models with comparable performance.

Inference cost: For the same performance, MoE inference computation is only 1/3 to 1/2 that of dense models, enabling lower API prices. For example, DeepSeek V4’s API price is only 1/10 of GPT‑4o’s, largely due to the cost advantage of MoE architecture.

Model upgrading: MoE has a major advantage not found in dense models: experts can be hot‑swapped. For instance, if you want to add capability in a specific domain (e.g., taxation), you don’t need to retrain the entire model—just train a single tax expert and add it. This is extremely important for enterprise customisation and domain fine‑tuning, costing only 1/10 of full fine‑tuning, which aids the adoption of mixture of experts models.

---

Challenges Facing Mixture of Experts Models

–Load‑balancing problem: If the gating network is poorly trained, it is easy for all tokens to be routed to a few popular experts while other experts are never invoked. The MoE then degenerates into a small dense model, with other experts occupying memory but receiving no tasks. Thus, training MoE requires adding a load‑balancing loss to force the gating network to distribute requests roughly evenly, but this slightly harms model performance.

–Low efficiency for small‑batch inference: MoE model excels in large‑batch, high‑concurrency scenarios: many requests arrive at once, keeping all experts busy with high utilisation. But in small‑batch, low‑concurrency settings (e.g., local deployment on personal devices), only a few requests come in each time, activating only a few experts; inference speed can be slower than that of a dense model with comparable performance, lowering cost‑effectiveness. MoE is better suited for high‑concurrency online API services, while dense models are more suitable for local deployment.

–Large memory footprint: Although only a subset of experts is activated each time, all expert weights must be loaded into memory, raising the deployment barrier. For comparison: a 70B dense model in FP16 needs 140GB of memory and can run on 2 A100 GPUs; DeepSeek V4 with 1.6T parameters in FP16 needs 3.2TB of memory, requiring at least 8 H100 GPUs to run. Consumer‑grade graphics cards can barely run flagship mixture of experts models. This is why MoE has only become more widely adopted with the proliferation of H100/H200 GPUs.

---

Innovations in Mixture of Experts (MoE) Technology

In recent years, important innovations have emerged in MoE architecture, aimed at further enhancing expert specialisation and overall model performance.

–Fine‑grained expert segmentation: Traditional MoE models use a relatively small number of large experts (e.g., Mixtral has 8). An intuitive idea: if we increase the number of experts while reducing their individual sizes, can we achieve better performance? This is the core idea of fine‑grained segmentation. By “cutting experts finer,” we can have more experts covering a wider range of specialised areas, increasing the potential for specialisation. For example, DeepSeek MoE 16B uses 64 fine‑grained experts. Studies show that with the same total and activated parameter counts, more fine‑grained experts generally yield performance improvements.

–Shared expert isolation: When processing natural language, some knowledge is general and fundamental (e.g., syntax, common vocabulary), while other knowledge is highly specialised (e.g., mathematical reasoning). If we force all experts to learn these general patterns, model capacity may be wasted and specialisation may be disrupted. To address this, DeepSeek proposed “Shared Experts”—experts that are always activated, independent of router decisions. They capture universal knowledge and low‑level features that the model must master. Other “Routed Experts” can then focus on learning more specialised, high‑level features. In the DeepSeek MoE architecture, the model’s output is the sum of the shared experts’ outputs and the selected routed experts’ outputs.

---

Application Scenarios For Mixture of Experts Models

–Handling large‑scale datasets: When the dataset is very large and diverse, MoE architecture assigns experts to different spatial scales and semantic categories, enabling efficient feature extraction from large‑scale complex data—e.g., in remote‑sensing image segmentation and point‑cloud semantic segmentation—significantly improving processing efficiency and model accuracy.

–Scalability for large models: MoE activates only a portion of experts during inference, reducing computational overhead, enabling faster inference and lower deployment costs, making it an effective approach for large language models (LLMs) and other complex tasks.

Recommended Related Reading from AI Robots Eidos

MoE is an architectural enhancement technique commonly introduced into VLA models to address the performance and efficiency bottlenecks they face in complex tasks, cross-modal generalization, and real-time inference. For example, MoE uses a Gating Network for dynamic routing, allowing different experts to specialize in different tasks (such as “grasping” or “obstacle avoidance”), thereby improving the cross-scenario generalization capability and task robustness of VLA.

Interested readers can refer to this article on VLA models:

Vision Language Action Models

–Need for model specialisation: MoE is highly effective when specific tasks or data subsets require specialised treatment. It allows the model to allocate experts to handle particular input types, e.g., complex legal text analysis, financial data mining, where different experts handle different feature dimensions and provide specialised advisory support.

–Pursuit of high‑efficiency computation: MoE models make computation more efficient through parallel processing and expert specialisation. Especially with high‑parallelism hardware like GPUs, MoE can greatly improve computational efficiency.

–Multi‑task learning: MoE is well‑suited for multi‑task learning, where different experts can be trained to handle different tasks, improving overall flexibility and efficiency. For instance, Google’s Gemini Ultra employs an MoE architecture that automatically routes queries to corresponding expert modules for different languages and domains; a single query can call in parallel both a translation expert and a coding expert.

---

Trends in Mixture of Experts Development

–Deep integration of MoE with inference optimisation: Techniques such as Speculative Decoding, KV Cache optimisation, and dynamic expert pruning are increasingly being combined with MoE. For example, at inference time, the model can dynamically adjust the number of activated experts based on task difficulty—simple tasks use only 1–2 experts, complex tasks use 4–6. Such adaptive inference computation is a key direction for future efficiency improvements.

–From “text MoE” to “multimodal MoE”: Currently, MoE model is mainly applied to pure‑text models. Multimodal scenarios (image + text + speech) are a natural fit for MoE—different modalities can be handled by different expert clusters, with the routing network determining whether the current input belongs to a particular modality or cross‑modal task. Google’s Gemini has already made substantial progress in this direction.

Image Credits: Generativeai & Linkedin & Datasciencedojo & Maartengrootendorst & Towardsai & Researchgate & Gurusup