What is Mixture of Experts in AI Models? Example - Qwen 3.5-122B-A10B

Mixture of Experts (MoE) in AI Models

Mixture of Experts (MoE) is a neural network architecture where instead of every token passing through a single dense feed-forward network, multiple expert subnetworks exist, and each token is sparsely routed to only a top-k subset of them. This lets models have massive total parameter counts while keeping inference compute proportional to a much smaller number of active parameters.

How MoE Works

An MoE layer replaces the standard FFN with two elements — a router/gate and an expert pool:

pythonCopy
# Conceptual MoE routing mechanism
import torch
import torch.nn as nn

class MixtureOfExperts(nn.Module):
    def __init__(self, d_model=4096, num_experts=64, top_k=8):
        super().__init__()
        self.num_experts = num_experts
        self.top_k = top_k

        self.router = nn.Linear(d_model, num_experts, bias=False)

        self.experts = nn.ModuleList([
            nn.Sequential(
                nn.Linear(d_model, d_model * 4),
                nn.SiLU(),
                nn.Linear(d_model * 4, d_model),
            ) for _ in range(num_experts)
        ])

    def forward(self, x):
        batch, seq, dim = x.shape
        x_flat = x.view(-1, dim)

        router_logits = self.router(x_flat)

        topk_weights, topk_indices = torch.topk(
            router_logits, self.top_k, dim=-1
        )
        topk_weights = torch.softmax(topk_weights, dim=-1)

        output = torch.zeros_like(x_flat)
        for k in range(self.top_k):
            expert_idx = topk_indices[:, k]
            weight = topk_weights[:, k]

            for e in range(self.num_experts):
                mask = (expert_idx == e)
                if mask.any():
                    output[mask] += weight[mask].unsqueeze(-1) *                                     self.experts[e](x_flat[mask])

        return output.view(batch, seq, dim)

Key Concepts

Qwen 3.5-122B-A10B Breakdown

This means Qwen achieves the knowledge capacity of a 122B model while only burning compute proportional to a ~10B dense model per forward pass.

MoE vs Dense Models

Popular MoE Models

Common Pitfalls

Key insight: MoE trades memory for compute efficiency. You need enough VRAM to hold all experts but only pay FLOPs for the active ones — which is why MoE models excel in throughput-constrained serving scenarios.

Learn more at Qwen Technical Report and Mixtral Paper.