Learning to Compose Neural Networks for Question Answering

By Jacob Andreas, Marcus Rohrbach, Trevor Darrell, Daniel Klein. 2016

584 citations recorded by the source corpus.

We describe a question answering model that applies to both images and structured knowledge bases. The model uses natural language strings to automatically assemble neural networks from a collection of composable modules. Parameters for these modules are learned jointly with network-assembly parameters via reinforcement learning, with only (world, question, answer) triples as supervision. Our approach, which we term a dynamic neural model network, achieves state-of-the-art results on benchmark datasets in both visual and structured domains.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

A Survey of Machine Learning for Big Code and Naturalness (2017, background): “roblem instance. For source code models, such architectures can derive its structure through static analyses. These architectures have been useful for visual question answering in NLP. Andreas et al. [17] create neural networks by composing them from “neural modules”, based on the input query struct”
Coarse-to-Fine Question Answering for Long Documents (2016, background): “Reinforcement learning recently gained popularity in tasks such as co-reference resolution (Clark and Manning, 2016), information extraction (Narasimhan et al., 2016), semantic parsing (Andreas et al., 2016) and textual games (Narasimhan et al., 2015; He et al., 2016).”
Break It Down: A Question Understanding Benchmark (2020, methodology): “QA using neural modular networks has been suggested for both KBs and images by Andreas et al. (2016) and Hu et al.”
ShapeWorld - A new test methodology for multimodal language understanding (2017, methodology): “In the near future, we plan to also adapt the technique of multimodal compact bilinear pooling (Fukui et al., 2016), neural module networks (Andreas et al., 2016a,b) and potentially also relation networks (Raposo et al., 2017) to the ICA task, and upload implementations to the GitHub repository.”
CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning (2016, methodology): “Some methods incorporate memory [42] or dynamic network architectures [2, 3].”
Programmable Agents (2017, background): “Neural Module Networks [2, 1], which use modularity in visual question answering, have been particularly inﬂuential on our thinking.”
Grammar-Based Grounded Lexicon Learning (2022, background): “Integrating symbolic structures such as programs and neural networks has shown success in modeling compositional queries in various domains, including image and video reasoning [20, 31], knowledge base query [4], and robotic planning [6].”
A Multi-Type Multi-Span Network for Reading Comprehension that Requires Discrete Reasoning (2019, methodology): “Later, Andreas et al. (2016a) proposed dynamically choosing an optimal layout structure from a list of layout candidates that are produced by off-the-shelf parsers.”
Unsupervised Question Decomposition for Question Answering (2020, neutral): “Examples include classical QA systems like SHRDLU (Winograd, 1972) and LUNAR (Woods et al., 1974), as well as neural Seq2Seq semantic parsers (Dong & Lapata, 2016) and neural module networks (Andreas et al., 2015; 2016).”
A Corpus for Reasoning about Natural Language Grounded in Photographs (2018, methodology): “We use two methods that do not require such formal programs: end-to-end neural module networks (N2NMN; Hu et al., 2017) and feature-wise linear modulation (F I LM; Perez et al., 2018).”
Neural Module Networks (2015, methodology): “We describe first steps toward joint learning of module behavior and a parser in a follow-up to this work [2].”
Neural Architecture Search with Reinforcement Learning (2016, methodology): “In machine learning, probabilistic program induction has been used successfully in many settings, such as learning to solve simple Q&A (Liang et al., 2010; Neelakantan et al., 2015; Andreas et al., 2016), sort a list of numbers (Reed & de Freitas, 2015), and learning with very few examples (Lake et ”
Ask Me Anything: Dynamic Memory Networks for Natural Language Processing (2015, background): “Another related model by Andreas et al. (2016) combines neural and logical reasoning for question answering over knowledge bases and visual question answering.”
Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding (2016, methodology): “We base our grounding approach on the fully-supervised version of GroundeR (Rohrbach et al., 2016).”
CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP (2021, background): “…(4) learning to accumulate knowledge and avoid catas- trophic forgetting in an continual learning setup (Jin et al., 2021); (5) decomposing complex tasks into atomic tasks and exploring cross-task general- ization through the lens of compositionality (An- dreas et al., 2016; Khot et al., 2021).”
Neural Programmer: Inducing Latent Programs with Gradient Descent (2015, background): “Later, dynamic neural module network (Andreas et al., 2016) was proposed for question answering which uses syntactic supervision in the form of dependency trees.”
Dynamic Memory Networks for Visual and Textual Question Answering (2016, background): “A model that also uses neural modules, albeit logically inspired ones, is that by Andreas et al. (2016) who evaluate on knowledgebase reasoning and visual question answering.”