Attention-Aware Routing (AAR) Boosts MoE Reasoning Without Retraining the Transformer

It is a routing method for Mixture-of-Experts models that uses temporal and spectral features from a sliding window of attention weights, rather than relying only on the token's hidden state, giving the router richer context.

Standard routing selects and weights experts based on the token's hidden state with limited contextual information, whereas AAR uses a summary of the model's contextual state derived from attention and disentangled from the hidden state.

The paper shows gains on OLMoE while training only routing parameters on a frozen transformer, making it a low-cost option for experimentation, but its depth sensitivity means layer selection must be tuned before production use.

Applying it indiscriminately across all layers can degrade factual retrieval, and current results are limited to OLMoE and GSM8K, so broader validation across models and tasks is needed.