MoE-LLaVA: Mixture of Experts for Large Vision-Language Models

By Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Jinfa Huang, Junwu Zhang, Munan Ning, Li Yuan. IEEE transactions on multimedia, 2024

360 citations recorded by the source corpus.

Recently, remarkable progress has been made in scaling up Large Language Models (LLMs) through the use of the sparse Mixture-of-Expert (MoE) layers without significantly increasing computational cost. However, the transition from a pre-trained LLM to a sparse Large Vision-Language Model (LVLM) with MoE remains an open challenge. Directly fine-tuning an LLM to a sparse LVLM often leads to training collapse, characterized by (1) a large modality feature distribution gap and (2) expert load imbalance. This paper proposes a three-stage decoupled weight training process. In the first two stages, the model learns to adapt the LLM to an LVLM. In the third stage, the FFN weights from the second stage are used as lossless initialization for expert weights, effectively constructing a sparse model with a vast number of parameters while maintaining constant computational cost. Through extensive ablation experiments, we derive three empirical guidelines and propose a sparse LVLM termed MoE-LLaVA. MoE-LLaVA is a MoE-based sparse LVLM architecture, which uniquely activates only the top-$k$ experts through routers during deployment, keeping the remaining experts inactive. Extensive experiments demonstrate that MoE-LLaVA outperforms LLaVA-1.5-7B by 4.6 on average across nine visual understanding benchmarks. Notably, MoE-LLaVA has only 2.2B active parameters, achieving an average performance of 66.7, surpassing LLaVA-1.5-7B’s 65.5. Our work establishes a baseline for sparse LVLMs and provides empirical guidelines for exploring the sparse LVLMs.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/cd1d7f5c4ce2d31ce9ee72db165a8272624da7d3

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

A Survey on Mixture of Experts in Large Language Models (2024, neutral): “Subsequently, Shen et al. [127] and Lin et al. [91] have further investigated the potential of MoE for scaling vision-language models, offering valuable insights that contribute to the development of more efﬁcient and effective multimodal learning systems.”
TinyVLA: Toward Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulation (2024, neutral): “…ﬁne-tuning technique LoRA [29], where the trainable parameters account for only 5% of the entire model; 3) We introduce a policy decoder that concatenated to pre-trained multimodal model through a simple but efﬁcient linear projection and output the executable action of the robot.”
How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suites (2024, neutral): “The development of open-source MLLMs [4, 15, 20, 43, 47, 49, 52, 65, 66, 98, 105, 113, 115, 119, 133, 134] has significantly influenced the AGI landscape by integrating and enhancing capabilities in processing both visual and textual data.”
What matters when building vision-language models? (2024, positive): “…2023; Sun et al., 2023; Radford et al., 2021) enables researchers to build upon these unimodal pre-trained models to create advanced VLMs that solve these problems with increasing accuracy (Dai et al., 2023; Liu et al., 2023; Bai et al., 2023; Lin et al., 2023; Li et al., 2023; Wang et al., 2023).”
LLaVA-Prumerge: Adaptive Token Reduction for Efficient Large Multimodal Models (2024, neutral): “Google’s Gemini [42] is a pioneer in efficient LMMs, achieving state-of-the-art performance on multi-modal benchmarks and introducing mobile-scale LMMs suitable for low-memory devices, although it is not open-source.”
Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding (2023, methodology): “In this work, we introduce Chat-UniVi, designed to not only comprehend and generate responses from text but also incorporate visual inputs, thereby providing a more comprehensive and immersive context for response generation.”