auxiliary-loss-free-balancing

Mixture of Experts Routing: Top-k Gating, Load Balancing and Aux-Loss-Free Balancing A sparse model is only as good as its router. Mixture-of-experts (MoE) layers promise a huge parameter count at…