Learning Conditioned Graph Structures for Interpretable Visual Question Answering

By Will Norcliffe-Brown, Efstathios Vafeias, Sarah Parisot. 2018

257 citations recorded by the source corpus.

Visual Question answering is a challenging problem requiring a combination of concepts from Computer Vision and Natural Language Processing. Most existing approaches use a two streams strategy, computing image and question features that are consequently merged using a variety of techniques. Nonetheless, very few rely on higher level image representations, which allow to capture semantic and spatial relationships. In this paper, we propose a novel graph-based approach for Visual Question Answering. Our method combines a graph learner module, which learns a question specific graph representation of the input image, with the recent concept of graph convolutions, aiming to learn image representations that capture question specific interactions. We test our approach on the VQA v2 dataset using a simple baseline architecture enhanced by the proposed graph learner module. We obtain state of the art results with 66.18% accuracy and demonstrate the interpretability of the proposed method.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Graph Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Graph Learning

Citations in the corpus

VisualBERT: A Simple and Performant Baseline for Vision and Language (2019, background): “Santoro et al. (2017), Norcliffe-Brown et al. (2018), and Cadene et al. (2019) explore using an attention module to implicitly model the relations between objects in the image.”
Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, background): “ work includes: (i) advanced attention mechanisms [40, 79, 51]; (ii) better multimodal fusion methods [14, 80, 27, 26]; (iii) multi-step reasoning [77, 22, 15]; (iv) incorporation of object relations [55, 48, 6, 32]; and (v) neural module networks for compositional reasoning [3, 25, 20, 11]. In prin”
Learning by Abstraction: The Neural State Machine (2019, methodology): “A few newer models use the relationships among objects to augment features with contextual information from each object’s surroundings [53, 75, 66].”
Structured Multimodal Attentions for TextVQA (2020, background): “The work [32] learns a question specific graph representation for input image in VQA, capturing object interactions with the relevant neighbours via spatial graph convolutions.”
Relation-Aware Graph Attention Network for Visual Question Answering (2019, background): “Some other work [50, 39, 52] have been proposed for learning question-conditioned graph representations for images.”
Language and Visual Entity Relationship Graph for Agent Navigation (2020, background): “In vision-and-language research, graph representations are often applied on objects in the scene [33, 26, 35, 14].”
Semantic Equivalent Adversarial Data Augmentation for Visual Question Answering (2020, methodology): “itional approaches [4,3,14], and bilinear pooling schemes [10,20]. Spatial attention [2] is one of the most widely used methods for both natural and synthetic image VQA. A large portion of prior arts [19,46,29,31] are built upon the bottom-up top-down (BUTD) attention method [2]. We also choose the ”
Answer Them All! Toward Universal Visual Question Answering Models (2019, methodology): “Algorithms apart from MAC and QCG demonstrate a large ( > 4 .”
Violin: A Large-Scale Dataset for Video-and-Language Inference (2020, unknown): “Many neural network models have been proposed for these tasks, such as more advanced attention mechanisms [64, 43, 70], better multimodal fusion methods [15, 71, 31, 30], the use of multi-step reasoning [24, 17, 7], the incorporation of relations [49, 38, 45], and neural module networks for composit”
Interpretable Visual Reasoning via Probabilistic Formulation Under Natural Supervision (2020, methodology): “In another research direction, to solve real-world problem, implicit reasoning via stacked attention modules [48,45,8,15] or graph reasoning [43,22,52,49,36] establish specially designed “black box” neural architectures.”
Reasoning with Heterogeneous Graph Alignment for Video Question Answering (2020, background): “So we have One thing to note is that similar work (Norcliffe-Brown, Vafeias, and Parisot 2018) targeting VQA tasks also utilized question conditioned GCN to deal with the relationships be-tween different visual regions but they did it simply by concatenating question embedding with each region’s…”
HyperGCN: A New Method of Training Graph Convolutional Networks on Hypergraphs (2018, background): “While explicit Laplacian regularisation assumes similarity among vertices in each edge / hyperedge, implicit regularisation of graph convolutional networks (GCNs) [25] avoids this restriction and enables application to a broader range of problems in combinatorial optimisation [19, 26, 38, 31], compu”
Iterative Context-Aware Graph Inference for Visual Dialog (2020, background): “[25] modeled a graph representation conditioned on the question, and exploited a novel graph convolution to capture the interactions among different detected object nodes.”
When Radiology Report Generation Meets Knowledge Graph (2020, methodology): “As many existing works (Yao et al. 2018; Liang et al. 2018; Chen et al. 2018; Norcliffe-Brown, Vafeias, and Parisot 2018; Hu et al. 2019), we used the graph convolution as a means of message passing and node interaction.”
Multi-Modal Graph Neural Network for Joint Reasoning on Vision and Scene Text (2020, methodology): “In addition, many interesting works [45, 36, 32, 21, 47] introduce GNN for grounding related task, such as referring expression [27] and visual question answering [54, 16, 23].”
SP-NAS: Serial-to-Parallel Backbone Search for Object Detection (2020, background): “Object detection is one of the core computer vision tasks to localize and classify multiple objects in an image and has been widely used in real-world applications [29, 28, 7, 2].”
Iterative Deep Graph Learning for Graph Neural Networks: Better and Robust Node Embeddings (2020, background): “Recently, researchers have explored methods to automatically construct a graph of objects [43, 8, 32, 14, 38] or words [37, 6, 7] when applying GNNs to non-graph structured data.”