DDRprog: A CLEVR Differentiable Dynamic Reasoning Programmer

By Joseph Suarez, Justin Johnson, Li Fei-Fei. 2018

33 citations recorded by the source corpus.

We present a generic dynamic architecture that employs a problem specific differentiable forking mechanism to leverage discrete logical information about the problem data structure. We adapt and apply our model to CLEVR Visual Question Answering, giving rise to the DDRprog architecture; compared to previous approaches, our model achieves higher accuracy in half as many epochs with five times fewer learnable parameters. Our model directly models underlying question logic using a recurrent controller that jointly predicts and executes functional neural modules; it explicitly forks subprocesses to handle logical branching. While FiLM and other competitive models are static architectures with less supervision, we argue that inclusion of program labels enables learning of higher level logical operations -- our architecture achieves particularly high performance on questions requiring counting and integer comparison. We further demonstrate the generality of our approach though DDRstack -- an application of our method to reverse Polish notation expression evaluation in which the inclusion of a stack assumption allows our approach to generalize to long expressions, significantly outperforming an LSTM with ten times as many learnable parameters.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

CLEVRER: CoLlision Events for Video REpresentation and Reasoning (2019, background): “There have also been numerous visual reasoning models (Hudson & Manning, 2018; Santoro et al., 2017; Hu et al., 2017; Perez et al., 2018; Zhu et al., 2017; Mascharka et al., 2018; Suarez et al., 2018; Cao et al., 2018; Bisk et al., 2018; Misra et al., 2018; Aditya et al., 2018).”
The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision (2019, methodology): “Table 1 compares our model with state-of-the-art visual reasoning models (Andreas et al., 2016; Suarez et al., 2018; Santoro et al., 2017) along four directions: visual features, semantics, inference, and the requirement of extra labels.”
Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding (2018, background): “These structures include the underlying functional programs [Mascharka et al., 2018, Suarez et al., 2018] and dependency trees [Cao et al., 2018] of the input question.”
Learning Visual Question Answering by Bootstrapping Hard Attention (2018, background): “DDRprog− [71], PG+EE (700k)− [70], TbD−, and TbD+hres− [50] are trained with a privileged state-description, while others are trained directly from images-questions-answers.”
Transparency by Design: Closing the Gap Between Performance and Interpretability in Visual Reasoning (2018, methodology): “Given the large number of highperforming models on CLEVR [14, 24, 5, 19], we train our model 5 times for a statistical measure of performance, achieving mean validation accuracy of 99.”
A Corpus for Reasoning about Natural Language Grounded in Photographs (2018, methodology): “Several methods have been proposed for compositional visual reasoning, including modular neural networks (e.g., Andreas et al., 2016; Johnson et al., 2017b; Perez et al., 2018; Hu et al., 2017; Suarez et al., 2018; Hu et al., 2018; Yao et al., 2018; Yi et al., 2018) and attention- or memory-based me”
VQA With No Questions-Answers Training (2018, methodology): “Originally modules arrangement was based on the dependency parsing of the question [6, 5], while later versions used supervised answering program learning [31, 26, 52, 63], including a probabilistic model [71].”