Where to Look: Focus Regions for Visual Question Answering
By Kevin J. Shih, Saurabh Singh, Derek Hoiem. 2015
487 citations recorded by the source corpus.
We present a method that learns to answer visual questions by selecting image regions relevant to the text-based query. Our method maps textual queries and visual features from various regions into a shared space where they are compared for relevance with an inner product. Our method exhibits significant improvements in answering questions such as "what color," where it is necessary to evaluate a specific location, and "what room," where it selectively identifies informative image regions. Our model is tested on the recently released VQA [1] dataset, which features free-form human-annotated questions and answers.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Question Answering & Reading Comprehension
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Computer Vision and Pattern Recognition
- Dacheng Tao
- Luc Van Gool
- Xiaogang Wang
- Trevor Darrell
- Ming-Hsuan Yang
- Bernt Schiele
- Yi Yang
- Jitendra Malik
In Question Answering & Reading Comprehension
- Anton van den Hengel
- Christopher Kanan
- Kushal Kafle
- Damien Teney
- Mohit Bansal
- Qi Wu
- Chunhua Shen
- Jie Lei
Citations in the corpus
- Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering (2016, methodology): “uilding trust for machines among their users. 2. Related Work Visual Question Answering. A number of recent works have proposed visual question answering datasets [3,21,25, 30,10,45,37,35] and models [9,24,2,42,23,26,46, 44,43,40,34,19,28,14,41,32,16]. Our work builds on top of the VQA dataset from ”
- Hierarchical Question-Image Co-Attention for Visual Question Answering (2016, background): “In [18], the authors generate image regions with object proposals and then select the regions relevant to the question and answer choice.”
- Deep Modular Co-Attention Networks for Visual Question Answering (2019, neutral): “The idea of learning visual attention on image regions from the input question in VQA was first proposed by [27, 7], and it becomes a de-facto component of almost all VQA approaches [10, 16, 1].”
- Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering (2015, methodology): “As noted in [29], the simple BOW model performs roughly as well if not better than the sequence-based LSTM for the VQA task.”
- Analyzing the Behavior of Visual Question Answering Models (2016, background): “A flurry of recent deep-learning based models have been proposed for VQA (Antol et al., 2015; Chen et al., 2015; Yang et al., 2016; Xu and Saenko, 2016; Jiang et al., 2015; Andreas et al., 2016a; Wang et al., 2015; Kafle and Kanan, 2016; Lu et al., 2016; Andreas et al., 2016b; Shih et al., 2016; Kim”
- MUTAN: Multimodal Tucker Fusion for Visual Question Answering (2017, background): “In [22], (image, question, answer) triplets are scored in an attentional framework.”
- Multimodal Explanations: Justifying Decisions and Pointing to the Evidence (2018, methodology): “Initial approaches to VQA used full-frame representations [22], but most recent approaches use some form of spatial attention [36, 35, 40, 8, 34, 30, 14, 18].”
- Dual Attention Networks for Multimodal Reasoning and Matching (2016, methodology): “A number of methods have recently adopted visual attention to benefit image classification [21, 28], image generation [6], image captioning [33], visual question answering [35, 26, 32], etc.”
- GuessWhat?! Visual Object Discovery through Multi-modal Dialogue (2016, background): “An extensive body of work has followed from this publication, largely building on the image captioning literature [3, 26, 37, 44].”
- Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering (2017, background): “has received a lot of attention – a number of VQA datasets have been curated [5, 21, 41, 12, 25, 11, 29, 13, 39] and a variety of deep-learning models have been developed [5, 9, 38, 37, 16, 3, 34, 19, 24, 4, 31, 20, 10, 26, 14, 35, 36, 40, 30].”
- TVQA+: Spatio-Temporal Grounding for Video Question Answering (2019, background): “One key to this success has been spatial attention [1, 34, 23], where neural models learn to attend to relevant regions for predicting the correct answer.”
- MarioQA: Answering Questions by Watching Gameplay Videos (2016, background): “On the other hand, image question answering (ImageQA) [3, 4, 5, 6, 10, 14, 22, 23, 24, 25, 28] addresses a holistic image understanding problem, and handles diverse recognition tasks in a single framework.”
- Beyond Bilinear: Generalized Multimodal Factorized High-Order Pooling for Visual Question Answering (2017, background): “Some approaches introduce off-the-shelf object detectors [16] or object proposals [23] as the candidates of”
- An Analysis of Visual Question Answering Algorithms (2017, background): “However, the annotations can be both non-exhaustive and duplicated, which makes using them to automatically make QA pairs difficult.”
- Attend Refine Repeat: Active Box Proposal Generation via In-Out Localization (2016, background): “Several approaches have been proposed in the literature for this task [1, 2, 5, 6, 8, 18, 25, 26, 29, 40, 44].”
- Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Input (2018, background): “…alignments in our case and other vision-and-language tasks such as captioning and VQA (which is often accomplished through an attention mechanism (Shih et al. 2015; Xu et al. 2015)) Zhou et al. (2016) demonstrate that global average pooling applied to the conv5 layer of several popular CNN…”
- Multi-modal Factorized Bilinear Pooling with Co-attention Learning for Visual Question Answering (2017, background): “Some approaches introduce off-the-shelf object detectors [9] or object proposals [27] as the attention region candidates and then use the question to identify related ones.”