Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts

By Basil Mustafa, Carlos Riquelme, Joan Puigcerver, Rodolphe Jenatton, Neil Houlsby. 2022

341 citations recorded by the source corpus.

Large sparsely-activated models have obtained excellent performance in multiple domains. However, such models are typically trained on a single modality at a time. We present the Language-Image MoE, LIMoE, a sparse mixture of experts model capable of multimodal learning. LIMoE accepts both images and text simultaneously, while being trained using a contrastive loss. MoEs are a natural fit for a multimodal backbone, since expert layers can learn an appropriate partitioning of modalities. However, new challenges arise; in particular, training stability and balanced expert utilization, for which we propose an entropy-based regularization scheme. Across multiple scales, we demonstrate remarkable performance improvement over dense models of equivalent computational cost. LIMoE-L/16 trained comparably to CLIP-L/14 achieves 78.6% zero-shot ImageNet accuracy (vs. 76.2%), and when further scaled to H/14 (with additional data) it achieves 84.1%, comparable to state-of-the-art methods which use larger custom per-modality backbones and pre-training schemes. We analyse the quantitative and qualitative behavior of LIMoE, and demonstrate phenomena such as differing treatment of the modalities and the organic emergence of modality-specific experts.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

A Review of Sparse Expert Models in Deep Learning (2022, methodology): “, 2020), whereas, the best performing models under transfer have used fewer, larger experts (Du et al., 2021; Zoph et al., 2022; Mustafa et al., 2022).”
Learning Customized Visual Models with Retrieval-Augmented Knowledge (2023, background): “There are an increasing number of studies to improve their generality from various modeling perspectives, including training objectives [11,13,14,38,65,68], scaling techniques [9,40,65], data efficiency [25, 28], and leveraging multilingual correlations [9, 19].”
MoE-LLaVA: Mixture of Experts for Large Vision-Language Models (2024, background): “Small-scale (million-level) models based on soft routers have also been explored in the context of multi-modal learning, such as EVE (Chen et al., 2023a) and LIMoE (Mustafa et al., 2022), which attempt a fusion of data by using soft routers.”
A Survey on Mixture of Experts in Large Language Models (2024, neutral): “One notable implementation of this approach is the LIMoE model [126], a sparse mixture of expert models tailored for multimodal learning.”
From Sparse to Soft Mixtures of Experts (2023, background): “Soft routing can provide stability while training a router; (Mustafa et al., 2022) observed that during training large fractions of input tokens can simultaneously change discrete routes through the network, leading to training challenges.”
Reproducible Scaling Laws for Contrastive Language-Image Learning (2022, background): “[65] are publicly available, enabling training language-vision models at larger scale [27, 49, 63].”
Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey (2023, methodology): “86 LIMoE [195] arXiv-2022 image-text Trans CS multi-modal pre-training with a sparse mixture of experts model 675M -”
Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints (2022, background): “Recently, there has been a growing number of MoE works achieving state-of-the-art quality and remarkable efficiency gains on both language and vision tasks (Lepikhin et al., 2021; Fedus et al., 2022; Riquelme et al., 2021; Artetxe et al., 2021; Du et al., 2021; Zoph et al., 2022; Mustafa et al., 202”
Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters (2024, background): “Based on the sparse architecture of MoE [63], some meth-ods [10, 20, 53, 62] are proposed to decrease computational costs and improve model capacity.”
Mixture-of-Experts in Remote Sensing: A Survey (2026, unknown): “In the multimodal setting, MoE backbones such as V-MoE [121], LIMoE [101] and Uni-Perceiver-MoE [158] combine image and text streams or more general modalities through shared Transformer [139] layers with MoE feed-forward blocks, where certain experts become specialized to modalities or sub-domains.”