Explicit Reasoning over End-to-End Neural Architectures for Visual Question Answering

By Somak Aditya, Yezhou Yang, Chitta Baral. 2018

55 citations recorded by the source corpus.

Many vision and language tasks require commonsense reasoning beyond data-driven image and natural language processing. Here we adopt Visual Question Answering (VQA) as an example task, where a system is expected to answer a question in natural language about an image. Current state-of-the-art systems attempted to solve the task using deep neural architectures and achieved promising performance. However, the resulting systems are generally opaque and they struggle in understanding questions for which extra knowledge is required. In this paper, we present an explicit reasoning layer on top of a set of penultimate neural network based systems. The reasoning layer enables reasoning and answering questions where additional knowledge is required, and at the same time provides an interpretable interface to the end users. Specifically, the reasoning layer adopts a Probabilistic Soft Logic (PSL) based engine to reason over a basket of inputs: visual relations, the semantic parse of the question, and background ontological knowledge from word2vec and ConceptNet. Experimental analysis of the answers and the key evidential predicates generated on the VQA dataset validate our approach.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

CLEVRER: CoLlision Events for Video REpresentation and Reasoning (2019, background): “COG (Yang et al., 2018) and MarioQA (Mun et al., 2017) use simulated environments to generate synthetic data and controllable reasoning tasks.”
Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding (2018, background): “Recently, Aditya et al. [2018] incorporated probabilistic soft logic into a neural attention module and obtained some interpretability of the model, and Gan et al. [2017] learned to associate image segments with questions.”
Right for the Right Concept: Revising Neuro-Symbolic Concepts by Interacting with their Explanations (2020, background): “research/NeSyXIL Several studies have investigated methods that produce explanations other than these visual explanations, such as multi-modal explanations [36, 54, 40], including visual and logic rule explanations [1, 39].”
Scene Graph Generation With External Knowledge and Image Reconstruction (2019, background): “There has been growing interest in improving data-driven models with external Knowledge Bases (KBs) in natural language processing [17, 4] and computer vision communities [24, 1, 6].”
VQA With No Questions-Answers Training (2018, methodology): “Some methods provide reasoning by using ”facts” extraction ( e.g . scene type) [39] or image caption results [28, 1, 27].”
RAVEN: A Dataset for Relational and Analogical Visual REasoNing (2019, background): “[1] proposed a probabilistic soft logic in an attention module to increase model interpretability.”