Learning to Reason: End-to-End Module Networks for Visual Question Answering

By Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, Kate Saenko. 2017

607 citations recorded by the source corpus.

Natural language questions are inherently compositional, and many are most easily answered by reasoning about their decomposition into modular sub-problems. For example, to answer “is there an equal number of balls and boxes?” we can look for balls, look for boxes, count them, and compare the results. The recently proposed Neural Module Network (NMN) architecture [3, 2] implements this approach to question answering by parsing questions into linguistic substructures and assembling question-specific deep networks from smaller modules that each solve one subtask. However, existing NMN implementations rely on brittle off-the-shelf parsers, and are restricted to the module configurations proposed by these parsers rather than learning them from data. In this paper, we propose End-to-End Module Networks (N2NMNs), which learn to reason by directly predicting instance-specific network layouts without the aid of a parser. Our model learns to generate network structures (by imitating expert demonstrations) while simultaneously learning network parameters (using the downstream task loss). Experimental results on the new CLEVR dataset targeted at compositional question answering show that N2NMNs achieve an error reduction of nearly 50% relative to state-of-theart attentional approaches, while discovering interpretable network architectures specialized for each question.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Break It Down: A Question Understanding Benchmark (2020, methodology): “QA using neural modular networks has been suggested for both KBs and images by Andreas et al. (2016) and Hu et al. (2017).”
LXMERT: Learning Cross-Modality Encoder Representations from Transformers (2019, methodology): “lenging visual reasoning dataset where some existing approaches (Hu et al., 2017; Perez et al., 2018) fail, and the SotA method is ‘MaxEnt’ in Suhr et al. (2019). The failure of existing methods (and our model w/o pre-training in Sec.”
Programmable Agents (2017, background): “This type of approach has also recently seen great success in answering questions that involve relational reasoning [15, 16].”
A simple neural network module for relational reasoning (2017, methodology): “46]. In our work, we follow a similar pipeline, but we use Relation Networks as a powerful reasoning module. Parallel to our work, two architectures have shown impressive results on the CLEVR dataset [14, 16]. Both approaches hinge on compositionality principles, and have shown they are capable of s”
Grammar-Based Grounded Lexicon Learning (2022, background): “Integrating symbolic structures such as programs and neural networks has shown success in modeling compositional queries in various domains, including image and video reasoning [18, 29], knowledge base query [4], and robotic planning [6].”
A Multi-Type Multi-Span Network for Reading Comprehension that Requires Discrete Reasoning (2019, background): “Hu et al. (2017) introduced an end-to-end module network that learns to predict instance-specific network layouts without the aid of a parser.”
VD-BERT: A Unified Vision and Dialog Transformer with BERT (2020, neutral): “Discriminative Setting Generative Setting MRR ↑ R@1 ↑ R@5 ↑ R@10 ↑ Mean ↓ MRR ↑ R@1 ↑ R@5 ↑ R@10 ↑ Mean ↓ LF (Das et HACAN (Yang et al., 2019), Synergistic (Guo et al., 2019a), DAN (Kang et al., 2019), and ReDAN (Gan et al., 2019).”
OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge (2019, background): “Modular networks [2, 18, 23] leverage the compositional nature of the language in deep neural networks.”
ViperGPT: Visual Inference via Python Execution for Reasoning (2023, background): “Posterior efforts have focused on explicitly reasoning about the composition by separating the reasoning from the perception, with connections to neuro-symbolic meth-ods [19, 27, 62].”
Visual Programming: Compositional visual reasoning without training (2022, methodology): “Neural module networks (NMN) [2] pioneered modular and compositional approaches for the visual question answering (VQA) task.”
FiLM: Visual Reasoning with a General Conditioning Layer (2017, methodology): “Some have argued that for artiﬁcial agents to learn this complex, structured process, it is necessary to build in aspects of reasoning, such as compositionality (Hu et al. 2017; Johnson et al. 2017b) or relational computation (Santoro et al. 2017).”
Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering (2016, background): “, NMN [5], N2NMN [19]) which learn various submodules for dif-”
A Corpus for Reasoning about Natural Language Grounded in Photographs (2018, methodology): “We use two methods that do not require such formal programs: end-to-end neural module networks (N2NMN; Hu et al., 2017) and feature-wise linear modulation (F I LM; Perez et al., 2018).”
Visual Entailment: A Novel Task for Fine-Grained Image Understanding (2019, methodology): “This includes simple linear models using ranking loss [16, 36], bi-linear pooling meth-ods [45, 20, 55, 17, 37], attention-based methods [1, 52, 64] and reasoning based approaches [54, 27, 33, 38, 29] on CLEVER and VQA-v1.”
Inferring and Executing Programs for Visual Reasoning (2017, background): “[16] propose end-to-end module networks for visual question answering which offer similar advantages.”
MAttNet: Modular Attention Network for Referring Expression Comprehension (2018, methodology): “Since their introduction, modular networks have been applied to several other tasks: visual reasoning [7, 11], question answering [2], relationship modeling [9], multitask reinforcement learning [1], etc.”