Visual7W: Grounded Question Answering in Images

By Yuke Zhu, Oliver Groth, Michael S. Bernstein, Li Fei-Fei. 2015

1,007 citations recorded by the source corpus.

We have seen great progress in basic perceptual tasks such as object recognition and detection. However, AI models still fail to match humans in high-level vision tasks due to the lack of capacities for deeper reasoning. Recently the new task of visual question answering (QA) has been proposed to evaluate a model's capacity for deep image understanding. Previous works have established a loose, global association between QA sentences and images. However, many questions and answers, in practice, relate to local regions in the images. We establish a semantic link between textual descriptions and image regions by object-level grounding. It enables a new type of QA with visual answers, in addition to textual answers used in previous work. We study the visual QA tasks in a grounded setting with a large collection of 7W multiple-choice QA pairs. Furthermore, we evaluate human performance and several baseline models on the QA tasks. Finally, we propose a novel LSTM model with spatial attention to tackle the 7W QA tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

Bidirectional Attention Flow for Machine Comprehension (2016, background): “At the coarse level of granularity, the question attends to different patches in the image (Zhu et al., 2016; Xiong et al., 2016a).”
VirTex: Learning Visual Representations from Textual Annotations (2020, background): “Vision-and-Language Pretraining attempts to learn joint representations of image-text paired data that can be transferred to multimodal downstream tasks such as visual question answering [13, 14, 59, 60], visual reasoning [61, 62], referring expressions [63], and language-based image retrieval [35].”
LXMERT: Learning Cross-Modality Encoder Representations from Transformers (2019, background): “, 2015), GQA balanced version (Hudson and Manning, 2019), and VG-QA (Zhu et al., 2016).”
Towards VQA Models That Can Read (2019, background): “VQA models typically use some variant of attention to get a representation of the image that is relevant for answering the question [2, 7, 30, 47, 48, 51, 17].”
How Much Can CLIP Benefit Vision-and-Language Tasks? (2021, methodology): “Following LXMERT, we use the same corpora aggregated from MS COCO Cap-tions (Chen et al., 2015), Visual Genome Cap-tions (Krishna et al., 2017), VQA (Antol et al., 2015), GQA (Hudson and Manning, 2019), and VG-QA (Zhu et al., 2016) for pre-training.”
STAR: A Benchmark for Situated Reasoning in Real-World Videos (2024, background): “Various benchmarks [11, 44, 20] have been constructed by asking annotators to provide designed questions [1, 44] or synthesizing questions with pre-defined templates [15, 24, 14, 6].”
Unifying Vision-and-Language Tasks via Text Generation (2021, methodology): “, 2019), GQA balanced version (Hudson & Manning, 2019), and Visual7W (Zhu et al., 2016)) as in Tan & Bansal (2019), but only used them for the visual question answering task.”
VL-BERT: Pre-training of Generic Visual-Linguistic Representations (2019, neutral): “l commonsense reasoning 3) masked visual-feature classiﬁcation 3) grounding referring expressions zLXMERT is pre-trained on COCO Caption (Chen et al., 2015), VG Caption (Krishna et al., 2017), VG QA (Zhu et al., 2016), VQA (Antol et al., 2015) and GQA (Hudson & Manning, 2019). Table 1: Compariso”
OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge (2019, background): “The idea of Visual 7W [47] is to provide object-level grounding for question-answer pairs as opposed to image-level associations between images and QA pairs.”
VizWiz Grand Challenge: Answering Visual Questions from Blind People (2018, methodology): “0: Real [17] MSCOCO [25] Crowd workers (AMT) Typed Visual7W [45] MSCOCO [25] Crowd workers (AMT) Typed CLEVR [19] Synthetic Shapes Automatically generated (templates) ——– SHAPES [6] Synthetic Shapes Automatically generated (templates) ——– Visual Genome [23] MSCOCO [25] & YFCC100M [35] Crowd workers ”
Visual Dialog (2016, background): “One key difference between VisDial and previous image question answering datasets (VQA [4], Visual 7W [62], Baidu mQA [12]) is the lack of a ‘visual priming bias’ in VisDial.”
Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering (2022, neutral): “Since the task of visual question answering (VQA) was ﬁrst proposed in [2], there have been plenty of VQA datasets [55, 57, 22, 10, 14, 11] conducted to facilitate the research work.”
Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering (2016, background): “For example, in another popular VQA dataset Visual7W [62], a question-only baseline achieves an accuracy of 46.”
Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering (2017, methodology): “Typically, these models can be characterized as top-down approaches, with context provided by a representation of a partially-completed caption in the case of image captioning [33, 27, 47, 45], or a representation of the question in the case of VQA [11, 28, 44, 46, 49].”
From Recognition to Cognition: Visual Commonsense Reasoning (2018, background): “This line of work also includes ‘pointing’ questions [45, 93] and templated questions with open ended answers [86].”
GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering (2019, background): “popular across the board, with a host of datasets being constructed [3, 9, 13, 36, 17] and numerous models being proposed [4, 33, 5, 8, 10].”
12-in-1: Multi-Task Vision and Language Representation Learning (2019, methodology): “Rel(vi, Q) = Wrhvi (3) Note thatQmay be either a phrase, question or dialog based on different tasks (RefCOCO+/g, Visual7W, GuessWhat).”