From Sparse to Soft Mixtures of Experts

By Joan Puigcerver, Carlos Riquelme, Basil Mustafa, Neil Houlsby. International Conference on Learning Representations, 2023

306 citations recorded by the source corpus.

Sparse mixture of expert architectures (MoEs) scale model capacity without significant increases in training or inference costs. Despite their success, MoEs suffer from a number of issues: training instability, token dropping, inability to scale the number of experts, or ineffective finetuning. In this work, we propose Soft MoE, a fully-differentiable sparse Transformer that addresses these challenges, while maintaining the benefits of MoEs. Soft MoE performs an implicit soft assignment by passing different weighted combinations of all input tokens to each expert. As in other MoEs, experts in Soft MoE only process a subset of the (combined) tokens, enabling larger model capacity (and performance) at lower inference cost. In the context of visual recognition, Soft MoE greatly outperforms dense Transformers (ViTs) and popular MoEs (Tokens Choice and Experts Choice). Furthermore, Soft MoE scales well: Soft MoE Huge/14 with 128 experts in 16 MoE layers has over 40x more parameters than ViT Huge/14, with only 2% increased inference time, and substantially better quality.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/2edccb8fa562ed52cd49ea6fc67ed32db6218247

Citations in the corpus

• A Survey on Mixture of Experts in Large Language Models (2024, unknown): “Oursurveyaimstoaddressthisdeficitbyprovidingaclearand comprehensiveoverviewofMoEinLLMs,introducinganovel taxonomythatorganizesrecentprogressintothreecategories: algorithm,system,andapplication.”
• An Empirical Study of the Non-Determinism of ChatGPT in Code Generation (2023, unknown): “The reason for such non-determinism with the temperature being zero is still controversial [1], with different hypotheses such as floating point, unreliable GPU calculations, and its sparse MoE architecture failing to enforce per-sequence determinism [44, 54].”