Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning

By Zhicheng Huang, Zhaoyang Zeng, Yupan Huang, Bei Liu, Dongmei Fu, Jianlong Fu. 2021

313 citations recorded by the source corpus.

We study joint learning of Convolutional Neural Network (CNN) and Transformer for vision-language pre-training (VLPT) which aims to learn cross-modal alignments from millions of image-text pairs. State-of-the-art approaches extract salient image regions and align regions with words step-by-step. As region-based visual features usually represent parts of an image, it is challenging for existing vision-language models to fully understand the semantics from paired natural languages. In this paper, we propose SOHO to "Seeing Out of tHe bOx" that takes a whole image as input, and learns vision-language representation in an end-to-end manner. SOHO does not require bounding box annotations which enables inference 10 times faster than region-based approaches. In particular, SOHO learns to extract comprehensive yet compact image features through a visual dictionary (VD) that facilitates cross-modal understanding. VD is designed to represent consistent visual abstractions of similar semantics. It is updated on-the-fly and utilized in our proposed pre-training task Masked Visual Modeling (MVM). We conduct experiments on four well-established vision-language tasks by following standard VLPT settings. In particular, SOHO achieves absolute gains of 2.0% R@1 score on MSCOCO text retrieval 5k test split, 1.5% accuracy on NLVR2 test-P split, 6.7% accuracy on SNLI-VE test split, respectively.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Scaling Up Vision-Language Pretraining for Image Captioning (2021, background): “More recently, instead of using object detectors for image feature extraction, end-to-end VLP based on convolution networks and transformers are becoming popular [14, 15, 19, 23, 44].”
Florence: A New Foundation Model for Computer Vision (2021, background): “Recently, there is an increasing trend (Huang et al., 2021; Xue et al., 2021; Wang et al., 2021; Kim et al., 2021; Dou et al., 2021) of end-to-end approaches to reduce dependency on the object bounding box, which instead consider grid-based feature representations as the fine-grained features for…”
FILIP: Fine-grained Interactive Language-Image Pre-Training (2021, methodology): “Recent efforts such as SOHO (Huang et al., 2021) and SimVLM (Wang et al.”
VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts (2021, background): “The second category models the interaction of images and text using a deep fusion encoder with cross-modal attention [43, 30, 41, 24, 51, 3, 26, 25, 14, 49, 16, 17, 20, 23, 46].”
SimVLM: Simple Visual Language Model Pretraining with Weak Supervision (2021, methodology): “To examine the quality of vision-language pretraining, we first compare SimVLM on the popular multi-modal tasks with state-of-the-art (SOTA) VLP methods including LXMERT (Tan & Bansal, 2019), VL-T5 (Cho et al., 2021), UNITER (Chen et al., 2020b), OSCAR (Li et al., 2020), Villa (Gan et al., 2020), SO”
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation (2021, background): “The first category focuses on modelling the interactions between image and text features with transformerbased multimodal encoders [10, 11, 12, 13, 1, 14, 15, 2, 3, 16, 8, 17, 18].”
UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning (2020, methodology): “Instead of extracting salient image regions by pre-trained object detection models like Faster-RCNN (Ren et al., 2016), the end-to-end vision-language pre-training architecture SOHO (Huang et al., 2021) proposes to jointly learn Convolutional Neural Network (CNN) and Transformer for cross-modal alig”
Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-training (2021, methodology): “Despite grid features overcome the restrictions of region-based image features and keep all visual information in images [18], CNN-based image feature learning remains challenging when utilized to vision-language pre-training.”
LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking (2022, methodology): “Image-text matching is widely used to learn a coarse-grained VL alignment [2, 5, 18, 21, 49].”
General Facial Representation Learning in a Visual-Linguistic Manner (2021, methodology): “Without relying on the the pre-trained object detector to extract image region features as previous works [17,43,56,90,104,105], CLIP [73] and ALIGN [44] adopt the contrastive loss, an effective loss in self-supervised representation learning, to pull the embedding of matched image-text pairs togeth”
FindIt: Generalized Localization with Natural Language Queries (2022, background): “Many approaches to vision-language learning leverage large-scale image-text pretraining or pre-computed detections [53,42,76,8,43,30,86,97,38,90,14,52,5,66].”
An Empirical Study of Training End-to-End Vision-and-Language Transformers (2021, methodology): “7 MLM+ITM SOHO [15] MLM+ITM+MIM CLIP-ViL [40] MLM+ITM+VQA SimVLM [50] 3 PrefixLM”
Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey (2023, methodology): “28 SOHO [153] CVPR-2021 image-text CNN+Trans MLM, MOR, MML Use a dynamic-updated visual dictionary for vision-language alignment URL”
VC-GPT: Visual Conditioned GPT for End-to-End Generative Vision-and-Language Pre-training (2022, methodology): “Evaluating on the image captioning downstream tasks, VC-GPT achieves either the best or the second-best performance across all evaluation metrics over the previous works, while only consuming 30 times less distinct images during cross-modal pre-training.”
Video-Text Pre-training with Learned Regions (2021, methodology): “In the past, some classic feature aggregation methods ( e.g. , VLAD [26], NetVLAD [4]) were used to construct a ﬁxed number of feature clusters to quantify image representation, and they have been proven effective in both image and video representation [23].”