Fusion of Detected Objects in Text for Visual Question Answering

By Chris Alberti, Jeffrey Ling, Michael Collins, David Reitter. 2019

185 citations recorded by the source corpus.

To advance models of multimodal context, we introduce a simple yet powerful neural architecture for data that combines vision and natural language. The “Bounding Boxes in Text Transformer” (B2T2) also leverages referential information binding words to portions of the image in a single unified architecture. B2T2 is highly effective on the Visual Commonsense Reasoning benchmark, achieving a new state-of-the-art with a 25% relative reduction in error rate compared to published baselines and obtaining the best performance to date on the public leaderboard (as of May 22, 2019). A detailed ablation analysis shows that the early integration of the visual features into the text analysis is key to the effectiveness of the new architecture. A reference implementation of our models is provided.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

UNITER: Learning UNiversal Image-TExt Representations (2019, methodology): “On the other hand, VisualBERT (Li et al., 2019b), Unicoder-VL (Li et al., 2019a), VL-BERT (Su et al., 2019) and B2T2 (Alberti et al., 2019) proposed the single-stream architecture, where a single Transformer is applied to both image and text.”
TAP: Text-Aware Pre-training for Text-VQA and Text-Caption (2020, background): “For the former, we reﬁne the pre-training tasks in VLP [37, 34] to support the extra scene text input.”
Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, methodology): “Compared to this two-stream architecture, recent work such as VL-BERT [60], VisualBERT [33], B2T2 [1], UnicoderVL [30] and UNITER [12] advocate a single-stream model design, where two modalities are directly fused in early stage.”
ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data (2020, methodology): “For example, most recent cross-modal pre-trained models [16, 17, 18, 19, 20, 21] use following 2 datasets in pre-training: The Conceptual Captions (CC) dataset[2], which contains 3M images with descriptions harvested from the Alt-text HTML attribute of the web pages, and SBU Captions[3], which consi”
MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, background): “[4] Chris Alberti, Jeffrey Ling, Michael Collins, and David Reitter.”
Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks (2021, methodology): “The most widely used objectives are image-text retrieval [2, 5, 43, 55, 63, 73, 74, 75], masked language modeling with image clues [2, 23, 43, 55, 72, 73, 74, 75], and masked region modeling [15, 55, 72, 73, 75].”
MERLOT: Multimodal Neural Script Knowledge Models (2021, background): “Most recently, A variety of papers have proposed “Visual BERT” style models [95, 17, 7, 58, 63, 107, 30]; these approaches train on datasets of image captions such as MSCOCO [59] and Visual Genome [50].”
Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts (2021, methodology): “Based directly upon BERT, V+L pre-training research has largely been focused on V+L understanding [52, 46, 21, 72, 3, 69, 45, 53], with classiﬁcation or regression tasks that do not involve generation.”
Hero: Hierarchical Encoder for Video+Language Omni-representation Pre-training (2020, neutral): “Compared to this two-stream architecture, VL-BERT (Su et al., 2019), UnicoderVL (Li et al., 2019a), B2T2 (Alberti et al., 2019), VisualBERT (Li et al., 2019b), and UNITER (Chen et al., 2019b) advocate single-stream architecture, where image and text signals are fused together in early stage.”
VD-BERT: A Unified Vision and Dialog Transformer with BERT (2020, methodology): “…such as ViLBERT (Lu et al., 2019) and LXMERT (Tan and Bansal, 2019), or a single-stream architecture to encode both text and image together, such as B2T2 (Alberti et al., 2019), Unicoder-VL (Li et al., 2019a), VisualBERT (Li et al., 2019b), VL-BERT (Su et al., 2019), and UNITER (Chen et al., 2019).”
Unified Vision-Language Pre-Training for Image Captioning and VQA (2019, neutral): “2019b), B2T2 (Alberti et al. 2019), Unicoder-VL (Li et al.”
VL-BERT: Pre-training of Generic Visual-Linguistic Representations (2019, background): “Compared with other concurrent works, i.e., ViLBERT, VisualBERT and B2T2, our VL-BERT achieves the state-of-the-art performance.”
Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers (2020, methodology): “On the other hand, there are some methods [1,6,18,22,29] apply single-stream neural network.”
12-in-1: Multi-Task Vision and Language Representation Learning (2019, background): “Multi-task V&L learning.”
Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning (2021, methodology): “Contrarily, the single-stream models use BERT [10] to learn a bi-directional joint distribution over the detection bounding box feature and text embedding feature [1, 7, 22, 23, 34, 48].”
Pre-Trained Models: Past, Present and Future (2021, methodology): “In single-stream models, such as VisualBERT (Li et al., 2019), Unicoder-VL (Li et al., 2020a), B2T2 (Alberti et al., 2019), the image region features and word embeddings are usually concatenated and fed into a single transformer.”
WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning (2021, methodology): “Multimodal models – such as ViLBERT [22], UNITER [6], Unicoder-VL [17] amongst others [1, 19, 32] – are able to jointly model the complex relationships between text and visual inputs leading to wins in downstream tasks like image search, Visual Question Answering (VQA) [2] and Visual Commonsense Rea”