Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
By Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Deepak Gotmare, Shafiq R. Joty, Caiming Xiong, Steven C. H. Hoi. 2021
2,847 citations recorded by the source corpus.
Large-scale vision and language representation learning has shown promising improvements on various vision-language tasks. Most existing methods employ a transformer-based multimodal encoder to jointly model visual tokens (region-based image features) and word tokens. Because the visual tokens and word tokens are unaligned, it is challenging for the multimodal encoder to learn image-text interactions. In this paper, we introduce a contrastive loss to ALign the image and text representations BEfore Fusing (ALBEF) them through cross-modal attention, which enables more grounded vision and language representation learning. Unlike most existing methods, our method does not require bounding box annotations nor high-resolution images. In order to improve learning from noisy web data, we propose momentum distillation, a self-training method which learns from pseudo-targets produced by a momentum model. We provide a theoretical analysis of ALBEF from a mutual information maximization perspective, showing that different training tasks can be interpreted as different ways to generate views for an image-text pair. ALBEF achieves state-of-the-art performance on multiple downstream vision-language tasks. On image-text retrieval, ALBEF outperforms methods that are pre-trained on orders of magnitude larger datasets. On VQA and NLVR$^2$, ALBEF achieves absolute improvements of 2.37% and 3.84% compared to the state-of-the-art, while enjoying faster inference speed. Code and pre-trained models are available at https://github.com/salesforce/ALBEF/.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jianfeng Gao
- Mohit Bansal
- Zhe Gan
- Caiming Xiong
- Percy Liang
- Yoshua Bengio
- Quoc V. Le
- Luke S. Zettlemoyer
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Ming-Hsuan Yang
- Xiaogang Wang
- Cordelia Schmid
- Andrew Zisserman
- Jitendra Malik
In Multimodal Learning
In Representation & Self-Supervised Learning
- Andrew Zisserman
- Cordelia Schmid
- Trevor Darrell
- Mohit Bansal
- Jianfeng Gao
- Andrea Vedaldi
- Tinne Tuytelaars
- Zhe Gan
Citations in the corpus
- Towards Language-Free Training for Text-to-Image Generation (2021, methodology): “Inspired by the BERT model [6], a number of methods attempt to learn generic multi-modal fusion layers, given the pre-extracted visual region features and textual encoder [21, 24, 26, 31, 43, 57].”
- Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, methodology): “, BERT, GPT-3, CLIP) have enabled impressive capabilities in recent years: from zero-shot image classification (Radford et al. 2021; Li et al. 2021a), to high-level planning (Huang et al.”
- Flamingo: a Visual Language Model for Few-Shot Learning (2022, background): “Another family of vision-language models is based on contrastive learning [2, 5, 49, 50, 57, 74, 82, 85, 138, 140, 146].”
- mSLAM: Massively multilingual joint pre-training for speech and text (2022, methodology): “…with SpanBERT (Joshi et al., 2020) and w2v-BERT (Chung et al., 2021) self-supervised losses that leverage unlabeled monomodal data, as well as a TLM loss (Conneau & Lample, 2019; Zheng et al., 2021) and a speech-text matching loss (Li et al., 2021) that both use supervised speech recognition data.”
- BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation (2022, methodology): “In this section, we compare BLIP to existing VLP methods on a wide range of vision-language downstream tasks2.”
- Scaling Up Vision-Language Pretraining for Image Captioning (2021, background): “More recently, instead of using object detectors for image feature extraction, end-to-end VLP based on convolution networks and transformers are becoming popular [14, 15, 19, 23, 44].”
- FILIP: Fine-grained Interactive Language-Image Pre-Training (2021, methodology): “(2) Another line of work (Li et al., 2021a; Kim et al., 2021) enforces the token-wise or patch-wise representations from both modalities into the same space and models these finer-grained interactions via cross-attention (Li et al., 2021a) or self-attention (Kim et al., 2021).”
- VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts (2021, methodology): “Global Hard Negative Mining Different from ALBEF [23], which samples hard negatives from training examples of the single GPU (named as local hard negative mining).”
- BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models (2023, neutral): “…proposed, including the dual-encoder architecture (Radford et al., 2021; Jia et al., 2021), the fusion-encoder architecture (Tan & Bansal, 2019; Li et al., 2021), the encoder-decoder architecture (Cho et al., 2021; Wang et al., 2021b; Chen et al., 2022b), and more recently, the unified…”
- GLIGEN: Open-Set Grounded Text-to-Image Generation (2023, background): “In contrast, in the recognition field, the long-standing paradigm has been to build recognition models [29, 37, 76] by starting from a foundation model pretrained on large-scale image data [4,15,16] or image-text pairs [30, 44, 68].”
- GIT: A Generative Image-to-text Transformer for Vision and Language (2022, background): “[62] Junnan Li, Ramprasaath R Selvaraju, Akhilesh Deepak Gotmare, Shafiq Joty, Caiming Xiong, and Steven Hoi.”
- ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models (2022, background): “This setting has been recently studied [32, 51, 33, 72], and is arguably more practical for real-world applications, as it brings the convenience towards the spirit of one-model-for-all.”
- Unified Contrastive Learning in Image-Text-Label Space (2022, background): “(i) Inspired by the success of BERT [13], the first line of research focuses on learning generic multi-modal fusion layers based on masked token prediction and/or image-text matching, given the pre-extracted features from visual and textual encoder [17, 31, 38, 39, 44, 51, 63, 73].”
- Generative Negative Text Replay for Continual Vision-Language Pretraining (2022, methodology): “For pretraining, we adopt a contrastive learning task to learn a generic and transferable visual-linguistic representation, which has proven to be efficient and effective in previous works [27].”
- CoCa: Contrastive Captioners are Image-Text Foundation Models (2022, methodology): “Since unidirectional language models are trained with causal masking on complete sentences, the decoder can efficiently generate outputs for both contrastive and generative losses with a single forward propagation (compared to two passes for a bidirectional approach [36]).”