Graph-Structured Representations for Visual Question Answering

By Damien Teney, Lingqiao Liu, Anton van den Hengel. 2016

452 citations recorded by the source corpus.

This paper proposes to improve visual question answering (VQA) with structured representations of both scene contents and questions. A key challenge in VQA is to require joint reasoning over the visual and text domains. The predominant CNN/LSTM-based approach to VQA is limited by monolithic vector representations that largely ignore structure in the scene and in the question. CNN feature vectors cannot effectively capture situations as simple as multiple object instances, and LSTMs process questions as series of words, which do not reflect the true complexity of language structure. We instead propose to build graphs over the scene objects and over the question words, and we describe a deep neural network that exploits the structure in these representations. We show that this approach achieves significant improvements over the state-of-the-art, increasing accuracy from 71.2% to 74.4% in accuracy on the abstract scenes multiple-choice benchmark, and from 34.7% to 39.1% in accuracy over pairs of balanced scenes, i.e. images with fine-grained differences and opposite yes/no answers to a same question.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Object-Centric Image Generation from Layouts (2020, methodology): “uction and matching tasks as the mapping from a scene graph to an image is one to many usually. These points explain their use in higher-level visual reasoning tasks such as visual question answering [56] and zero-shot learning [54], and also motivate the use of scene graph-based retrieval in our mo”
GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering (2019, background): “Other works have argued for the need to devise new evaluation measures and techniques to shed more light on systems’ inner workings [16, 29, 30, 15].”
Tips and Tricks for Visual Question Answering: Learnings from the 2017 Challenge (2017, background): “This can also be seen as a special case of highway networks [29] and has been mentioned in other work in natural language processing [11, 30].”
Scene Graph Generation by Iterative Message Passing (2017, methodology): “However, these models that use scene graphs either rely on groundtruth annotations [18], synthetic images [37], or extract a scene graph from text domain [1, 4].”
Auto-Encoding Scene Graphs for Image Captioning (2018, methodology): “ch includes the knowledge of present objects, their attributes, and pairwise relationships. Thus, the scene graph can provide a beneﬁ- cial prior for other vision tasks like image retrieval [13], VQA [39], and image generation [11]. By observing the potential of exploiting scene graphs in vision tas”
Neural Motifs: Scene Graph Parsing with Global Context (2017, background): “Predicting such graph representations has been shown to improve natural language based image tasks [17, 38, 46] and has the potential to signiﬁcantly expand the scope of applications for computer vision systems.”
Unbiased Scene Graph Generation From Biased Training (2020, neutral): “Scene graph generation (SGG) [65] — a visual detection task of objects and their relationships in an image — seems to have never fulfilled its promise: a comprehensive visual scene representation that supports graph reasoning for high-level tasks such as visual captioning [70, 68] and VQA [57, 15].”
Learning by Abstraction: The Neural State Machine (2019, methodology): “A few newer models use the relationships among objects to augment features with contextual information from each object’s surroundings [53, 75, 66].”
Graph R-CNN for Scene Graph Generation (2018, background): “These scene graphs form an interpretable structured representation of the image that can support higher-level visual intelligence tasks such as captioning [24,39], visual question answering [1,11,35,37–39], and image-grounded dialog [3].”
Structure Inference Net: Object Detection Using Scene-Level Context and Instance-Level Relationships (2018, background): “Several interesting works [28, 34, 23, 39, 21, 9, 2, 22, 42] have been proposed to combine deep networks with graphical models for structured prediction tasks that are solved by structure inference techniques.”
Dynamic Multimodal Instance Segmentation guided by natural language queries (2018, background): “…of Computer Vision (CV) and Natural Language Understanding (NLU) is an active area of research that includes multiple tasks such as object detection based on natural language expressions [9][10], image captioning [11][12][13][14] and visual question answering (VQA) [15][16][17][18][19].”
Relation-Aware Graph Attention Network for Visual Question Answering (2019, methodology): “[50] exploited structured question representations such as parse trees, and used GRU to model contextualized interactions between both objects and words.”
Structured Multimodal Attentions for TextVQA (2020, background): “[27] construct graphs over image scene objects and over”
Multimodal Graph Networks for Compositional Generalization in Visual Question Answering (2020, methodology): “Teney et al. [47] use graphs to represent both images and text but merely combine the representations using a weighted sum.”
Language and Visual Entity Relationship Graph for Agent Navigation (2020, background): “[33] build graphs over the scene objects and over the question words, and exploit the structures in these representations for visual question answering.”
ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning (2022, background): “In future, we will develop better representations including semantic graph representations (Teney et al., 2017) that can exploit the relations among the question, chart objects, and data values.”