VL-BERT: Pre-training of Generic Visual-Linguistic Representations

By Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, Jifeng Dai. 2019

1,848 citations recorded by the source corpus.

We introduce a new pre-trainable generic representation for visual-linguistic tasks, called Visual-Linguistic BERT (VL-BERT for short). VL-BERT adopts the simple yet powerful Transformer model as the backbone, and extends it to take both visual and linguistic embedded features as input. In it, each element of the input is either of a word from the input sentence, or a region-of-interest (RoI) from the input image. It is designed to fit for most of the visual-linguistic downstream tasks. To better exploit the generic representation, we pre-train VL-BERT on the massive-scale Conceptual Captions dataset, together with text-only corpus. Extensive empirical analysis demonstrates that the pre-training procedure can better align the visual-linguistic clues and benefit the downstream tasks, such as visual commonsense reasoning, visual question answering and referring expression comprehension. It is worth noting that VL-BERT achieved the first place of single model on the leaderboard of the VCR benchmark. Code is released at \url{this https URL}.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

UNITER: Learning UNiversal Image-TExt Representations (2019, methodology): “On the other hand, VisualBERT (Li et al., 2019b), Unicoder-VL (Li et al., 2019a), VL-BERT (Su et al., 2019) and B2T2 (Alberti et al., 2019) proposed the single-stream architecture, where a single Transformer is applied to both image and text.”
BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation (2021, background): “unfair outcomes and negative experiences to the end users of the downstream applications [11, 18, 20, 33, 34].”
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks (2019, background): “More recent preprints including VLBERT [48], Unified VLP [49] and UNITER [49] also show promising improvements in this research direction of joint visio-linguistic pretraining.”
Contrastive Learning of Medical Visual Representations from Paired Images and Text (2020, methodology): “• Contrastive-Binary-Loss : This baseline diﬀers from ConVIRT by contrasting the paired image and text representations with a binary classiﬁcation head, as is widely done in visual-linguistic pretraining work (Tan and Bansal, 2019; Su et al., 2020).”
ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph (2020, background): “…by pre-trained models like BERT [1] and ERNIE [2] which have signiﬁcantly improved the performance on many NLP tasks, researchers ( [3] [4] [5] [6] [7] [8] [9] [10]) have noticed the importance of pre-training for vision-language tasks, e.g., Visual Question Answering(VQA) [11] and Visual…”
Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, methodology): “Compared to this two-stream architecture, recent work such as VL-BERT [60], VisualBERT [33], B2T2 [1], UnicoderVL [30] and UNITER [12] advocate a single-stream model design, where two modalities are directly fused in early stage.”
Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks (2020, background): “Recent studies [23,39,5,36,20,19,47] on vision-language pretraining (VLP) have shown it effective to learn generic representations from massive image-text pairs, and that fine-tuning VLP models on task-specific data achieves state-of-the-art (SoTA) results on well-established V+L tasks.”
ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data (2020, methodology): “For example, most recent cross-modal pre-trained models [16, 17, 18, 19, 20, 21] use following 2 datasets in pre-training: The Conceptual Captions (CC) dataset[2], which contains 3M images with descriptions harvested from the Alt-text HTML attribute of the web pages, and SBU Captions[3], which consi”
Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training (2019, methodology): “VL-BERT masked the image before applying by Faster-RCNN.”
Towards Language-Free Training for Text-to-Image Generation (2021, methodology): “Inspired by the BERT model [6], a number of methods attempt to learn generic multi-modal fusion layers, given the pre-extracted visual region features and textual encoder [21, 24, 26, 31, 43, 57].”
CLIP-GEN: Language-Free Training of a Text-to-Image Generator with CLIP (2022, background): “Vision-Language Modelling Pre-training methods that have recently moved from raw text to multi-modal data (e.g., image-text) have revolutionized numerous multimodal tasks (e.g., image-text matching [9, 27, 34], image captioning [44], Visual Question Answering [9,20,33,34]).”
How Much Can CLIP Benefit Vision-and-Language Tasks? (2021, methodology): “…V&L pre-training has been proposed as an effective technique to improve the performance on various V&L tasks (Lu et al., 2019; Tan and Bansal, 2019; Li et al., 2019a; Su et al., 2019; Chen et al., 2020; Zhou et al., 2020; Huang et al., 2020; Li et al., 2020; Zhang et al., 2021; Li et al., 2021b).”
StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery (2021, background): “Joint representations Multiple works learn cross-modal Vision and language (VL) representations [8, 39, 44, 29, 24, 43, 23, 4, 26] for a variety of tasks, such as languagebased image retrieval, image captioning, and visual question answering.”
Flamingo: a Visual Language Model for Few-Shot Learning (2022, background): “[106] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai.”