E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning

By Haiyang Xu, Ming Yan, Chenliang Li, Bin Bi, Songfang Huang, Wenming Xiao, Fei Huang. 2021

130 citations recorded by the source corpus.

Vision-language pre-training (VLP) on large-scale image-text pairs has achieved huge success for the cross-modal downstream tasks. The most existing pre-training methods mainly adopt a two-step training procedure, which firstly employs a pre-trained object detector to extract region-based visual features, then concatenates the image representation and text embedding as the input of Transformer to train. However, these methods face problems of using task-specific visual representation of the specific object detector for generic cross-modal understanding, and the computation inefficiency of two-stage pipeline. In this paper, we propose the first end-to-end vision-language pre-trained model for both V+L understanding and generation, namely E2E-VLP, where we build a unified Transformer framework to jointly learn visual representation, and semantic alignments between image and text. We incorporate the tasks of object detection and image captioning into pre-training with a unified Transformer encoder-decoder architecture for enhancing visual learning. An extensive set of experiments have been conducted on well-established vision-language downstream tasks to demonstrate the effectiveness of this novel VLP paradigm.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Achieving Human Parity on Visual Question Answering (2021, methodology): “To address the limitations of region-based features like locality, some works such as PixelBERT [29], E2E-VLP [87], and Grid-VLP [88] have been proposed to revisit grid-based convolutional features for multi-modal learning, skipping the expensive region-related steps.”
SimVLM: Simple Visual Language Model Pretraining with Weak Supervision (2021, background): “attribute prediction (Xu et al., 2021), contrastive loss (Li et al.”
OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework (2022, background): “Multimodal Pretraining Multimodal pretraining has been developing rapidly [38, 13, 39, 40, 14, 41, 42, 43, 44, 15, 16, 17, 45, 46, 47].”
Towards General Purpose Vision Systems: An End-to-End Task-Agnostic Vision-Language Architecture (2021, background): “E2E-VLP [53] presents an end-to-end trainable architecture that is extensively pretrained with masked language modeling, image-text matching, captioning, and object detection objectives, each with a different output head.”
Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts (2021, methodology): “They either utilize pre-trained object detectors (Tan and Bansal, 2019; Lu et al., 2019; Li et al., 2019, 2020a; Chen et al., 2020; Li et al., 2020c; Gan et al., 2020) or conduct object detection on-thefly in the pre-training process (Su et al., 2019; Xu et al., 2021).”
X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrieval (2022, background): “With the success of self-supervised pre-training such as BERT [12] in NLP, vision-language pre-training on large-scale unlabeled cross-modal data has attracted growing attention [23, 31–33, 37, 44, 50, 51, 55, 60].”
Multimodal Learning With Transformers: A Survey (2022, background): “recently include random sampling [113], learning concept dictionary [203], uniform masking [270], patch projection [192], joint learning of a region detector [271], and representation aligning before mask prediction [263].”
Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey (2023, background): “The aligned image and text are capable of training a model for text generation based on a given image, for example, Xu et al. train the E2E-VLP [125] with ITG objective: where X represents the visual sequence with context, Y denotes the generated set of text, and the length of tokens in text y is n…”
mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality (2023, methodology): “In this paper, we present mPLUG-Owl with an innovative modularized training paradigm for large multi-modal language models that can support multiple modalities concurrently, drawing inspiration from the concept of modularization [Xu et al., 2023b, Li et al., 2022, Xu et al., 2021, Ye et al., 2022].”
Webly Supervised Concept Expansion for General Purpose Vision Models (2022, background): “[84] Haiyang Xu, Ming Yan, Chenliang Li, Bin Bi, Songfang Huang, Wenming Xiao, and Fei Huang.”
KD-VLP: Improving End-to-End Vision-and-Language Pretraining with Object Knowledge Distillation (2021, methodology): “Pretraining Corpus: Following the E2E pre-training strategy (Huang et al., 2021, 2020; Xu et al., 2021), we take indomain datasets: MSCOCO (Lin et al., 2014) and VG (Krishna et al., 2016) as pretraining datasets since it is widely used in literature.”
FindIt: Generalized Localization with Natural Language Queries (2022, background): “Many approaches to vision-language learning leverage large-scale image-text pretraining or pre-computed detections [53,42,76,8,43,30,86,97,38,90,14,52,5,66].”
UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling (2021, methodology): “UniTAB achieves better performance than prior arts [68,13] that use similar amounts of pre-training images, with the CIDEr score improved from 117.”
Universal Captioner: Long-Tail Vision-and-Language Model Training through Content-Style Separation (2021, methodology): “While several works have encoded regions through additive attention [22, 25, 41], graph-based encodings [16, 63, 65], or self-attentive structures [9, 21, 32, 39], the emergence of self-attentive visual encoders [12] and large-scale multi-modal models like CLIP [42] is reopening the discussion on wh”
mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections (2022, methodology): “To improve the inference speed, some recent work such as Pixel-BERT [3], E2E-VLP [19] and ViLT [10] removes the complicated object detector in feature extraction, and conducts end-toend VL learning with CNN-based grid features and linearly projected patched embeddings, respectively.”
Injecting Semantic Concepts into End-to-End Image Captioning (2021, methodology): “This procedure can be expressed as follows: Several VL models [28,33,72] obtain a great improvement in inference speed by using general image encoders without regional operations.”