Inferring and Executing Programs for Visual Reasoning

By Justin Johnson, Bharath Hariharan, L. V. D. van der Maaten, Judy Hoffman, Li Fei-Fei, C. Lawrence Zitnick, Ross B. Girshick. 2017

586 citations recorded by the source corpus.

Existing methods for visual reasoning attempt to directly map inputs to outputs using black-box architectures without explicitly modeling the underlying reasoning processes. As a result, these black-box models often learn to exploit biases in the data rather than learning to perform visual reasoning. Inspired by module networks, this paper proposes a model for visual reasoning that consists of a program generator that constructs an explicit representation of the reasoning process to be performed, and an execution engine that executes the resulting program to produce an answer. Both the program generator and the execution engine are implemented by neural networks, and are trained using a combination of backpropagation and REINFORCE. Using the CLEVR benchmark for visual reasoning, we show that our model significantly outperforms strong baselines and generalizes better in a variety of settings.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Prompting & Reasoning

Citations in the corpus

Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, background): “multimodal fusion methods [14, 80, 27, 26]; (iii) multi-step reasoning [77, 22, 15]; (iv) incorporation of object relations [55, 48, 6, 32]; and (v) neural module networks for compositional reasoning [3, 25, 20, 11]. In principle, our proposed VILLA framework can be plugged into these “shallower” mo”
Programmable Agents (2017, background): “This type of approach has also recently seen great success in answering questions that involve relational reasoning [15, 16].”
A simple neural network module for relational reasoning (2017, methodology): “Parallel to our work, two architectures have shown impressive results on the CLEVR dataset [14, 16].”
Unsupervised Question Decomposition for Question Answering (2020, methodology): “We use FAISS (Johnson et al., 2017a) to efﬁciently build S (cid:48) .”
STAR: A Benchmark for Situated Reasoning in Real-World Videos (2024, methodology): “Language Parser Language Parser parses each question to a functional program [18, 41] in the form of a program sentence.”
OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge (2019, background): “Modular networks [2, 15, 19] leverage the compositional nature of the language in deep neural networks.”
ViperGPT: Visual Inference via Python Execution for Reasoning (2023, background): “[15] and other work in this direction [12, 14, 31] typically implement π with a neural network that is trained with either supervised or reinforcement learning in order to estimate programs from queries.”
A Corpus for Reasoning about Natural Language Grounded in Photographs (2018, methodology): “CLEVR (Johnson et al., 2017a), where the language is synthetically generated.”
Visual Entailment: A Novel Task for Fine-Grained Image Understanding (2019, methodology): “This includes simple linear models using ranking loss [16, 36], bi-linear pooling meth-ods [45, 20, 55, 17, 37], attention-based methods [1, 52, 64] and reasoning based approaches [54, 27, 33, 38, 29] on CLEVER and VQA-v1.”
Learning to Reason: End-to-End Module Networks for Visual Question Answering (2017, methodology): “Our model is different from [16] in that we use a set of specialized modules with soft attention mechanism to provide textual parameters for each module, while [16] uses a generic mod-Question How many other things are of the same size as the green matte ball?”
CLEVRER: CoLlision Events for Video REpresentation and Reasoning (2019, methodology): “…study the problem of temporal and causal reasoning in videos from a complementary perspective: inspired by a recent visual reasoning dataset, CLEVR (Johnson et al., 2017a), we simplify the problem of visual recognition, but emphasize the complex temporal and causal structure behind the interacting…”
Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions (2018, background): “…playing games (Denil et al., 2017; Kansky et al., 2017), learning algorithms (Bošnjak et al., 2017; Cai et al., 2017; Li et al., 2016; Reed & De Freitas, 2015), visual understanding (Ellis et al., 2017; Johnson et al., 2017), and natural language processing (Andreas et al., 2016; Hu et al., 2017).”
Psycholinguistics Meets Continual Learning: Measuring Catastrophic Forgetting in Visual Question Answering (2019, methodology): “This shows that despite the limited output space of the Y/N-q task, such type of questions in CLEVR are complex and require reasoning skills (Johnson et al., 2017a).”
The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision (2019, methodology): “The SOTA methods do not provide interpretable representation on individual objects (Johnson et al., 2017a; Hudson & Manning, 2018; Mascharka et al., 2018) .”
Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding (2018, methodology): “NS-VQA outperforms the IEP baseline under various conditions, even with a weaker supervision during REINFORCE (2K and 9K question-answer pairs in REINFORCE).”