Scaling Laws for Fine-Grained Mixture of Experts

By Jakub Krajewski, Jan Ludziejewski, Kamil Adamczewski, Maciej Pi'oro, Michal Krutul, Szymon Antoniak, Kamil Ciebiera, Krystian Kr'ol, Tomasz Odrzyg'o'zd'z, Piotr Sankowski and 2 more. 2024

Subfields: Large Language Models, Model Architectures, Scaling & Learning Theory

Citations in the corpus

• Training Sparse Mixture Of Experts Text Embedding Models (2025, unknown): “This aligns with findings from Krajewski et al. (2024), who observed that MoE models tend to underperform dense models under limited training regimes.”