Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering

By Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, Lei Zhang. 2017

4,718 citations recorded by the source corpus.

Top-down visual attention mechanisms have been used extensively in image captioning and visual question answering (VQA) to enable deeper image understanding through fine-grained analysis and even multiple steps of reasoning. In this work, we propose a combined bottom-up and top-down attention mechanism that enables attention to be calculated at the level of objects and other salient image regions. This is the natural basis for attention to be considered. Within our approach, the bottom-up mechanism (based on Faster R-CNN) proposes image regions, each with an associated feature vector, while the top-down mechanism determines feature weightings. Applying this approach to image captioning, our results on the MSCOCO test server establish a new state-of-the-art for the task, achieving CIDEr / SPICE / BLEU-4 scores of 117.9, 21.5 and 36.9, respectively. Demonstrating the broad applicability of the method, applying the same approach to VQA we obtain first place in the 2017 VQA Challenge.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Experience Grounds Language (2020, background): “Or the 1,600 classes of Anderson et al. (2017). Torchvision/Detectron2 include dozens of trained models.”
UNITER: Learning UNiversal Image-TExt Representations (2019, methodology): “Figure 2 lists the splits from VQA, Image-Text Retrieval, COCO Captioning, Ref-COCO/RefCOCO+/RefCOCOg, and the bottom-up top-down (BUTD) detection (Anderson et al., 2018), all from COCO images.”
VisualBERT: A Simple and Performant Baseline for Vision and Language (2019, methodology): “…al., 2015; Goyal et al., 2017), textual grounding (Kazemzadeh et al., 2014; Plummer et al., 2015), and visual reasoning (Suhr et al., 2019; Zellers et al., 2019) have been proposed and various models (Yang et al., 2016; Anderson et al., 2018; Jiang et al., 2018) have been developed to solve them.”
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks (2019, methodology): “The latter mimics common attention mechanisms found in vision-and-language models [30].”
TAP: Text-Aware Pre-training for Text-VQA and Text-Caption (2020, background): “Text-Caption seeks to generate an image caption [54, 4] that describes both the visual and scene text information in the image, as shown in Figure 1 (a).”
ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph (2020, methodology): “Following [34], we treat VQA as a multi-label classiﬁcation task – assigning a soft target score to each answer based on its relevancy to the 10 human answer responses.”
VirTex: Learning Visual Representations from Textual Annotations (2020, methodology): “As described in Section 2, all these methods use an object detector trained on Visual Genome [73] (with ImageNet-pretrained backbone) to extract visual features, made available by [74].”
Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, methodology): “Image and text inputs are first represented as low-dimensional feature vectors zimg = gbu(ximg) and ztxt = gemb(xtxt), where gbu(·) represents a fixed bottom-up image feature extractor [2], and gemb(·) represents a learnable word embedding function.”
Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks (2020, methodology): “odal semantic alignment from scratch, and thus improves the learning eciency. In fact, our base model already outperforms the existing large VLP models on most V+L tasks. Object Tags Anderson et al. [2] introduce the bottom-up mechanism to represent an image as a set of visual regions via Faster R-C”
ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data (2020, background): “Image Captioning[9] aims to generate a natural language description for each input image.”
LXMERT: Learning Cross-Modality Encoder Representations from Transformers (2019, methodology): “Model Architecture: Our model is closely related to three ideas: bi-directional attention, Trans-former, and BUTD.”
Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training (2019, background): “We will move further to evaluate it on more cross-modal tasks, such as image captioning (Anderson et al. 2018), scene graph generation, video classiﬁcation and video question answering.”
Towards VQA Models That Can Read (2019, methodology): “For images, the visual features are represented as spatial features, either in the form of grid-based convolutions and/or features extracted from the bounding box proposals [1].”
How Much Can CLIP Benefit Vision-and-Language Tasks? (2021, methodology): “CLIP-ViL p with CLIP-Res50 outperforms models based on the widely used region-based encoder, BottomUp-TopDown (BUTD) ResNet101 (Anderson et al., 2018a).”
LILA: Language-Informed Latent Actions (2021, methodology): “This is enabled by our [ Right ] language-informed latent actions (LILA) models that use auto-encoders to learn language & state-conditioned low-DoF latent spaces for meaningful control. robotics settings for similar purposes within the full autonomy paradigm [9, 10, 11, 12, 13].”
MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, methodology): “Nonetheless, UNITER-Base has 154 million parameters, though some are frozen (86 million from their Transformer, 23 million from the word embedding layer, and then 44 million from their object detector [5]).”
VL-ADAPTER: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks (2021, methodology): “ViLBERT [32], UNITER [6] and LXMERT [48] use the faster R-CNN [43] object detector to extract the bottom-up [1] features from the image and provide them with text features to a cross-modality transformer to solve visual question answering and image-text alignment tasks.”
VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling (2021, methodology): “Different from MRM and MFM, which rely on distilled visual categories or features from a well-supervised visual backbone [41, 45], we present Masked Visual-token Modeling (MVM) to perform masked visual modeling in a self-reconstruction scenario.”
Florence: A New Foundation Model for Computer Vision (2021, methodology): “Thus, the object detector has been a de facto tool for image feature extraction, followed by a fusion network for prediction in many works (Anderson et al., 2018; Li et al., 2020; Zhang et al., 2021b; Wang et al., 2020; Fang et al., 2021; Chen et al., 2020d).”