ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data

By Di Qi, Lin Su, Jianwei Song, Edward Cui, Taroon Bharti, Arun Sacheti. 2020

280 citations recorded by the source corpus.

In this paper, we introduce a new vision-language pre-trained model -- ImageBERT -- for image-text joint embedding. Our model is a Transformer-based model, which takes different modalities as input and models the relationship between them. The model is pre-trained on four tasks simultaneously: Masked Language Modeling (MLM), Masked Object Classification (MOC), Masked Region Feature Regression (MRFR), and Image Text Matching (ITM). To further enhance the pre-training quality, we have collected a Large-scale weAk-supervised Image-Text (LAIT) dataset from Web. We first pre-train the model on this dataset, then conduct a second stage pre-training on Conceptual Captions and SBU Captions. Our experiments show that multi-stage pre-training strategy outperforms single-stage pre-training. We also fine-tune and evaluate our pre-trained ImageBERT model on image retrieval and text retrieval tasks, and achieve new state-of-the-art results on both MSCOCO and Flickr30k datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Transferable Visual Models From Natural Language Supervision (2021, background): “…demonstrated an additional set of (image, text) pairs collected from the internet could improve retrieval performance and several new automatically constructed datasets such as Conceptual Captions (Sharma et al., 2018), LAIT (Qi et al., 2020), and OCR-CC (Yang et al., 2020) have been created.”
Learning Visual Representations with Caption Annotations (2020, background): “r free. For instance, user tags associated with images can be used as image labels [32,41]. Even richer, companion text for images, is sometimes available for free. Using recent sanitation procedures [48], high-quality large-scale captioned datasets can automatically be constructed. In this paper, w”
ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph (2020, methodology): “Most of them ( [5] [6] [7] [8] [9] [20]) use a uniform cross-modal Transformer modelling both image and text representations, while the others like ViLBERT [4] and LXMERT [21] are based on two-stream cross-modal Transformers, which bring more specific representations for image and text.”
Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks (2020, methodology): “We did not use ranking losses [14,18], as we found that the binary classification loss works better, as reported in [28].”
FILIP: Fine-grained Interactive Language-Image Pre-Training (2021, methodology): “We compare our FILIP model against methods with complex attention layers including Unicoder-VL (Li et al., 2020a), ImageBERT (Qi et al., 2020), UNITER (Chen et al., 2020), VILLA (Gan et al., 2020), ERNIE-ViL (Yu et al., 2021), Oscar (Li et al., 2020b), VinVL (Zhang et al., 2021), ALBEF (Li et al.,…”
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation (2021, background): “The ﬁrst category focuses on modelling the interactions between image and text features with transformer-based multimodal encoders [10, 11, 12, 13, 1, 14, 15, 2, 3, 16, 8, 17, 18].”
VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning (2021, methodology): “Unlike neural networks pretrained on multimodal data (e.g., [39,48,57,66,67,82,83]), our method only requires a small amount of multimodal training data and focuses on adapting linguistic knowledge learned from the textual modality.”
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision (2021, methodology): “With ﬁne-tuning, ALIGN outperforms all existing methods by a large margin, including those that employ more complex cross-modal attention layers such as ImageBERT (Qi et al., 2020), UNITER (Chen et al., 2020c), Table 1.”
ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision (2021, methodology): “The pretrain-and-finetune scheme has been expanded to a joint domain of vision and language, giving birth to the category of Vision-and-Language Pretraining (VLP) models (Lu et al., 2019; Chen et al., 2019; Su et al., 2019; Li et al., 2019; Tan & Bansal, 2019; Li et al., 2020a; Lu et al., 2020; Qi e”
Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers (2020, methodology): “To learn a universal representation for most vision and language tasks, we ﬁrst pre-train our model with image-sentence pair dataset similar to other cross-modality pre-training researches [6,18,19,21,22,29,33,41].”
Pre-Trained Models: Past, Present and Future (2021, methodology): “ImageBERT (Qi et al., 2020) is the same as Unicoder-VL.”
WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning (2021, methodology): “Similarly, models like VL-BERT [32], VisualBERT [19], ImageBERT [25], B2T2 [1] and Unicoder-VL [18], all pretrain on CC or similar datasets using a variety of objectives and tasks.”
X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers (2020, methodology): “…transformer architecture to jointly encode text and image (Li et al., 2019; Su et al., 2019; Al-berti et al., 2019; Rahman et al., 2020; Li et al., 2020; Chen et al., 2019; Qi et al., 2020; Huang et al., 2020), while others use two-stream architectures (Lu et al., 2019, 2020; Tan and Bansal, 2019).”
WuDaoCorpora: A super large-scale Chinese corpora for pre-training language models (2021, background): “The quality of NLP models strongly relies on the scale of corpus used for model training (He et al., 2018; Duan et al., 1912; Cui et al., 2019; Qi et al., 2001).”
Video Transformer for Deepfake Detection with Incremental Learning (2021, methodology): “A large number of transformer based models are used to deal with the vision tasks, such as image classi � cation[22], object detection [8], image segmentation [49], image captioning [34], video classi � cation [24, 48], and visual question answering [11, 34, 42].”
Transferring Knowledge from Vision to Language: How to Achieve it and how to Measure it? (2021, methodology): “ViLBERT, LXMERT, ImageBERT and OSCAR (Lu et al., 2019; Tan and Bansal, 2019; Qi et al., 2020; Li et al., 2020b).”
WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research (2023, background): “…the counterpart of AL multimodal learning, has been studied for decades and has contributed to the success of many vision-language applications, such as cross-modal search [22], image and video captioning [23], [24], text-to-image generation [25], [26], and visual question answering [27], [28].”