VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts

By Wenhui Wang, Hangbo Bao, Li Dong, Furu Wei. 2021

762 citations recorded by the source corpus.

We present a unified Vision-Language pretrained Model (VLMo) that jointly learns a dual encoder and a fusion encoder with a modular Transformer network. Specifically, we introduce Mixture-of-Modality-Experts (MoME) Transformer, where each block contains a pool of modality-specific experts and a shared self-attention layer. Because of the modeling flexibility of MoME, pretrained VLMo can be fine-tuned as a fusion encoder for vision-language classification tasks, or used as a dual encoder for efficient image-text retrieval. Moreover, we propose a stagewise pre-training strategy, which effectively leverages large-scale image-only and text-only data besides image-text pairs. Experimental results show that VLMo achieves state-of-the-art results on various vision-language tasks, including VQA, NLVR2 and image-text retrieval. The code and pretrained models are available at https://aka.ms/vlmo.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Flamingo: a Visual Language Model for Few-Shot Learning (2022, background): “In particular, BERT [23] inspired a large body of vision-language work [16, 28, 29, 38, 59, 61, 66, 101, 106, 107, 109, 118, 121, 142, 143, 151].”
LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, background): “Apart from textual instructions, LLaMA-Adapter is capable of answering a question based on input of other modalities, which augments the language model with rich cross-modal information.”
Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models (2023, methodology): “Following the instruction of Transformer [39], pre-trained LLMs demonstrate a powerful text understanding and generation capability [31, 19, 37, 5], and such breakthroughs also benefit the VL modelling [13, 14, 3, 49], where these works add an extra adapter modules [17] in the pre-trained LLMs to al”
BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models (2023, neutral): “…converged to a few time-tested ones: image-text contrastive learning (Radford et al., 2021; Yao et al., 2022; Li et al., 2021; 2022), image-text matching (Li et al., 2021; 2022; Wang et al., 2021a), and (masked) language modeling (Li et al., 2021; 2022; Yu et al., 2022; Wang et al., 2022b).”
Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks (2022, methodology): “In this work, we adopt Multiway Transformers [WBDW21] for general-purpose modeling, i.e., one uniﬁed architecture shared for various downstream tasks.”
OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework (2022, methodology): “Besides, to simplify preprocessing, patch projection has received attention and helped Transformer achieve SOTA performance in downstream tasks [22, 48].”
data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language (2022, background): “There has been a considerable body of research on learning representations of multiple modalities simultaneously often using paired data (Aytar et al., 2017; Radford et al., 2021b; Wang et al., 2021; Singh et al., 2021) with the aim to produce cross-modal representations which can perform well on mu”
Language Models are General-Purpose Interfaces (2022, methodology): “We follow the evaluation pipeline proposed in FLAN (Wei et al., 2021).”
Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts (2021, neutral): “We notice that some concurrent work (Wang et al., 2021a) also obtains superior performance on visual reasoning tasks, while X-VLM is still SoTA at comparable settings.”
Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks (2022, methodology): “Many models have been proposed for tasks with text and/or image input and text output (Cho et al., 2021; Wang et al., 2022d; Li et al., 2022b; Wang et al., 2021; Kaiser et al., 2017; Sun et al., 2022; Chen et al., 2022d; Wang et al., 2022c).”
MoE-LLaVA: Mixture of Experts for Large Vision-Language Models (2024, background): “A series of works (Bao et al., 2022; Long et al., 2023; Satar et al., 2022; Wang et al., 2022; Shen et al., 2023) naturally decouple experts based on modal categories and pre-define each expert to handle a specific modality.”
Multimodal biomedical AI (2022, background): “…is, a unique task activates only a small part of the network, with the model learning which parts of the network should handle each unique task) 110 , produce aligned representations for similar concepts across modalities (for example, the picture of a dog, and the word ‘dog’ should produce…”
Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey (2023, methodology): “35 VLMo [156] arXiv-2021 image-text Trans MLM, MML, CS Jointly learns visual-, text-encoder and a fusion encoder URL”
VLP: A Survey on Vision-language Pre-training (2022, background): “Recently, VDBERT[109] modelled the common implicit vision-language alignment in vision and language by pre-training on large-scale image-text pairs via transfer learning[110, 111].”
PromptCap: Prompt-Guided Task-Aware Image Captioning (2022, background): “Recent studies [31,58,64,68] further explore combining single modality and fusion encoders with a modular network design.”
mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections (2022, methodology): “Most methods [1, 20, 4, 7] deal with visual question answering tasks as multi-label classification on predefined answer sets.”
A Survey on Vision--Language--Action Models for Embodied AI (2024, neutral): “For a more comprehensive comparison of human datasets, we refer interested readers to [217].”