Optimizing MoE Routing with Expert-Space Exploration

Expert-Space Exploration Reinforcement Learning, or ESRL, treats the routing mechanism of a Mixture-of-Experts model as the primary lever for reinforcement learning. By selectively perturbing which experts are involved during training, this framework increases exploration diversity while keeping the model on computationally reliable paths. This approach improves performance on mathematics, science, and coding tasks without requiring additional sampling overhead or changes to the model's fundamental architecture.

Expert Path Selection

ESRL identifies anchor experts by tracking the top-k highest probability routing outputs for a given input. When a token is processed, the system calculates the probability distribution across all experts. Experts with routing weights that reach a specific confidence threshold are designated as anchors and kept fixed in the computational path. The remaining capacity is filled by selecting from the pool of non-anchor experts. The framework determines this selection by measuring the entropy of the router's distribution. As the router's confidence fluctuates, the system adjusts the inclusion criteria for secondary experts.

This method allows the model to explore alternative routing paths that were not the primary choice of the original frozen gate but are nonetheless plausible candidates for processing the token. By sampling these candidates during rollout, the model gathers data on which expert configurations actually produce correct answers for difficult reasoning tasks. If a specific path leads to a correct answer, the reinforcement learning process increases the probability that the router will select that path when it encounters similar inputs in the future.

Mechanism and Implementation

To bridge the gap between exploration and policy updates, ESRL tracks the routing decisions made during rollout and updates the router's weights to favor configurations that yield high rewards. When the model generates an output, the system evaluates it against the expected task outcome, such as the correct solution to a math problem. If the answer is correct, the system uses the reward as a signal to adjust the routing logits. The gradient of this reward is backpropagated to the router's weights for the non-anchor experts that participated in that successful generation. This process shifts the logit distribution, making those successful expert configurations more likely to be selected the next time the router encounters similar tokens.

This approach remains compatible with diverse MoE backbones, including top-K, top-1, and shared-expert routing structures. It is not limited to flat architectures, meaning it can be applied to most common MoE implementations without requiring significant structural modifications. The primary advantage here is that the model learns to refine its internal allocation policy directly, rather than relying on temperature-based sampling of the final output distribution which often produces incoherent or redundant text.

Performance and Known Unknowns

For the Qwen3-30B-A3B model, this method yields a 3.2 percentage point improvement in Pass@1 and a 4.5 point gain in Pass@8. These gains are significant because they are achieved without increasing the number of samples taken per token during training.

A concrete example of why this works involves multi-step reasoning. In a standard MoE, the router may default to a specific set of experts that are good at general language modeling but fail at complex logic. By forcing the inclusion of diverse, non-anchor experts during training, the model can discover that a different, niche set of experts actually performs better on logical reasoning. Once that path is identified as high-reward, the router's weights are permanently adjusted to prioritize that path.

What remains unknown is the long-term impact on expert utilization. We do not yet have data on whether the anchor-selection bias eventually leads to expert collapse, a state where the model loses its ability to utilize the full breadth of its parameter count because it over-optimizes for a narrow set of pre-defined anchor configurations. Additionally, we do not know if these gains will scale to much larger MoE models with thousands of experts, or if the entropy-based exploration will eventually favor suboptimal noise when the router is forced to select from a pool of experts that have no relevance to the input token. These questions will determine whether ESRL is a general-purpose improvement or a technique reserved for specific reasoning-heavy tasks.