Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers

By Zhicheng Huang, Zhaoyang Zeng, Bei Liu, Dongmei Fu, Jianlong Fu. 2020

475 citations recorded by the source corpus.

We propose Pixel-BERT to align image pixels with text by deep multi-modal transformers that jointly learn visual and language embedding in a unified end-to-end framework. We aim to build a more accurate and thorough connection between image pixels and language semantics directly from image and sentence pairs instead of using region-based image features as the most recent vision and language tasks. Our Pixel-BERT which aligns semantic connection in pixel and text level solves the limitation of task-specific visual representation for vision and language tasks. It also relieves the cost of bounding box annotations and overcomes the unbalance between semantic labels in visual task and language semantic. To provide a better representation for down-stream tasks, we pre-train a universal end-to-end model with image and sentence pairs from Visual Genome dataset and MS-COCO dataset. We propose to use a random pixel sampling mechanism to enhance the robustness of visual representation and to apply the Masked Language Model and Image-Text Matching as pre-training tasks. Extensive experiments on downstream tasks with our pre-trained model show that our approach makes the most state-of-the-arts in downstream tasks, including Visual Question Answering (VQA), image-text retrieval, Natural Language for Visual Reasoning for Real (NLVR). Particularly, we boost the performance of a single model in VQA task by 2.17 points compared with SOTA under fair comparison.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

TAP: Text-Aware Pre-training for Text-VQA and Text-Caption (2020, methodology): “For the former, we reﬁne the pre-training tasks in VLP [37, 34] to support the extra scene text input.”
ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph (2020, methodology): “Most of them ( [5] [6] [7] [8] [9] [20]) use a uniform cross-modal Transformer modelling both image and text representations, while the others like ViLBERT [4] and LXMERT [21] are based on two-stream cross-modal Transformers, which bring more specific representations for image and text.”
How Much Can CLIP Benefit Vision-and-Language Tasks? (2021, methodology): “…V&L pre-training has been proposed as an effective technique to improve the performance on various V&L tasks (Lu et al., 2019; Tan and Bansal, 2019; Li et al., 2019a; Su et al., 2019; Chen et al., 2020; Zhou et al., 2020; Huang et al., 2020; Li et al., 2020; Zhang et al., 2021; Li et al., 2021b).”
VL-ADAPTER: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks (2021, methodology): “Pix-elBERT [18] replaces R-CNN with plain ResNet [14], and demonstrates the advantage of including the vision model in training.”
Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks (2021, methodology): “[24] attempts to eliminate the need for object detectors by directly extracting features from the raw pixels with CNNs. [1, 22, 28, 56] take a further step by applying Transform-ers to raw image patches and word tokens.”
Scaling Up Vision-Language Pretraining for Image Captioning (2021, background): “More recently, instead of using object detectors for image feature extraction, end-to-end VLP based on convolution networks and transformers are becoming popular [14, 15, 19, 23, 44].”
Combined Scaling for Zero-shot Transfer Learning (2021, neutral): “…Hodosh et al., 2013; Frome et al., 2013; Norouzi et al., 2013; Kiros et al., 2014; Socher et al., 2014; Akata et al., 2015b,a; Nam et al., 2017; Faghri et al., 2017; Li et al., 2019; Liu et al., 2019; Lu et al., 2019; Messina et al., 2020; Chen et al., 2020d; Huang et al., 2020; Chen et al., 2021).”
UFO: A UniFied TransfOrmer for Vision-Language Representation Learning (2021, methodology): “The image can be encoded as object features [5,6,13,18,27,29,30,30,32,39,42,47,53, 54] through a Faster RCNN [36], as grid features [20, 22] from a convolutional neural network, or as patch features [25] from a linear projection on raw pixels.”
VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts (2021, background): “The second category models the interaction of images and text using a deep fusion encoder with cross-modal attention [43, 30, 41, 24, 51, 3, 26, 25, 14, 49, 16, 17, 20, 23, 46].”
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision (2021, methodology): “We evaluate the zero-shot model performance of ALIGN and compare it with M 3 P (Huang et al., 2020a) and UC2 (Zhou et al., 2021).”
Unifying Vision-and-Language Tasks via Text Generation (2021, methodology): “Following this success, image+text pretraining models (Lu et al., 2019; Tan & Bansal, 2019; Chen et al., 2020; Huang et al., 2020; Li et al., 2020b; Cho et al., 2020; Radford et al., 2021; Zhang et al., 2021) and video+text pretraining models (Sun et al., 2019b;a; Li et al., 2020a; Zhu & Yang,…”
Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers (2021, methodology): “Though Huang et al. (2020) showed a detector can be replaced with an image classifier, we show that comparable results can be achieved without any image-only pretraining.”
Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, neutral): “Several recent methods [5, 16, 19, 34, 47, 49, 51, 55, 56, 73, 76, 101] pretrain multi-modal vision-language representations, such as transformers, using datasets with image cap-tions, e.g., COCO [15], Conceptual Captions [70] and Visual Genome [41].”
Hero: Hierarchical Encoder for Video+Language Omni-representation Pre-training (2020, methodology): “More recently, ViLBERT is enhanced by multi-task learning (Lu et al., 2019b), Oscar (Li et al., 2020) enhances pre-training with image tags, and Pixel-BERT (Huang et al., 2020) proposes to align image pixels (instead of bottomup features (Anderson et al., 2018)) with text.”
GIT: A Generative Image-to-text Transformer for Vision and Language (2022, background): “Recently, a growing interest is in dense representation (Huang et al., 2020; Wang et al., 2021b;a; Kim et al., 2021; Fang et al., 2021b; Dou et al., 2021; Li et al., 2021a) from the feature map, which requires no bounding box annotations.”
ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision (2021, methodology): “The pretrain-and-finetune scheme has been expanded to a joint domain of vision and language, giving birth to the category of Vision-and-Language Pretraining (VLP) models (Lu et al., 2019; Chen et al., 2019; Su et al., 2019; Li et al., 2019; Tan & Bansal, 2019; Li et al., 2020a; Lu et al., 2020; Qi e”