Bottom-Up and Top-Down Attention for Image Captioning and VQA

By Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, Lei Zhang. 2017

357 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Women also Snowboard: Overcoming Bias in Captioning Models (2018, background): “, [30,31,32,33,34]) is to increase overall sentence quality.”
VizWiz Grand Challenge: Answering Visual Questions from Blind People (2018, methodology): “VQA In the main paper, we report results for the algorithm proposed in [1].”
FiLM: Visual Reasoning with a General Conditioning Layer (2017, methodology): “This approach differs from classical visual question answering pipelines which fuse image and language information into a single embedding via element-wise product, concatenation, attention, and/or more advanced methods (Yang et al. 2016; Lu et al. 2016; Anderson et al. 2017).”
GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering (2019, methodology): “To verify the reliability of the metric, we further perform experiments with spatial features instead of the object-informed ones used by BottomUp [4] and MAC [10], which lead to a much lower 43% score, demonstrating that object-based features provide models with better granularity for the task, all”
Bottom-Up Abstractive Summarization (2018, methodology): “Anderson et al. (2017) develop a state-of-the-art model with a two-step approach that ﬁrst pre-computes bounding boxes of segmented objects and then applies attention to these regions.”
Stacked Cross Attention for Image-Text Matching (2018, background): “[1] to analogize the detection of salient regions at object/stuff level to the spontaneous bottom-up attention in the human vision system [4, 6, 21], and practically imple-”
TVQA: Localized, Compositional Video Question Answering (2018, methodology): “For simplicity, we follow (Anderson et al., 2018; Karpathy and Fei-Fei, 2015) selecting the top-K regions1 from each detected label across all frames.”
Object-Driven Text-To-Image Synthesis via Adversarial Training (2019, background): “In 2018, [1] proposes a bottom-up attention mechanism, which enables attention to be calculated over semantic meaningful regions/objects in the image, for image captioning and visual question-answering.”
Tips and Tricks for Visual Question Answering: Learnings from the 2017 Challenge (2017, methodology): “We refer to this stage as the top-down attention, as opposed to the model of Anderson et al .. [3] that provides image features from bottom-up attention.”
TVQA+: Spatio-Temporal Grounding for Video Question Answering (2019, methodology): “Temporal and Spatial Attention Attention has shown great success on many vision and language tasks, such as image captioning [1, 40], visual question answering [1, 36], language grounding [42], etc.”
GQA: a new dataset for compositional question answering over real-world images (2019, methodology): “Beside these, we evaluate the performance of the bottom-up attention model [4] – the winner of 2017 VQA challenge, and the MAC model [11] – a compositional attention state-of-the-art model for CLEVR [15].”
Grounded Video Description (2018, methodology): “For image captioning, we compare against two SotA methods, Neural Baby Talk (NBT) [15] and BUTD [1].”
Object Hallucination in Image Captioning (2018, methodology): “The attention mechanism from (Anderson et al., 2018) can be used on such feature maps, but Anderson et al.”
Learning to Count Objects in Natural Images for Visual Question Answering (2018, methodology): “…we extract the attention weights of the ﬁrst attention glimpse (there are two in the baseline) before softmax normalization, and feed them into the counting component after Anderson et al. (2017) vary from 10 to 100 per image, a natural choice for the number of top-n proposals to use is 10.”
A Comprehensive Survey of Deep Learning for Image Captioning (2018, methodology): “[4] applied both top down and bottom up approaches.”
Guide Me: Interacting with Deep Networks (2018, background): “Most works focus on the combination of CNN and RNN models, often building attention mechanisms [3, 42, 59, 60].”
Transparency by Design: Closing the Gap Between Performance and Interpretability in Visual Reasoning (2018, background): “Some approaches [1, 4, 18, 26, 27, 28] propose an attention mechanism whereby each word corresponds to some feature of an image.”
Learning by Abstraction: The Neural State Machine (2019, methodology): “Typically, most approaches use either CNNs [83, 86] or object detectors [5] to derive visual features which are then compared to a ﬁxed-size question embedding obtained by an LSTM.”