Neural Module Networks

By Jacob Andreas, Marcus Rohrbach, Trevor Darrell, Daniel Klein. 2015

1,208 citations recorded by the source corpus.

Visual question answering is fundamentally compositional in nature-a question like where is the dog? shares substructure with questions like what color is the dog? and where is the cat? This paper seeks to simultaneously exploit the representational capacity of deep networks and the compositional linguistic structure of questions. We describe a procedure for constructing and learning neural module networks, which compose collections of jointly-trained neural "modules" into deep networks for question answering. Our approach decomposes questions into their linguistic substructures, and uses these structures to dynamically instantiate modular networks (with reusable components for recognizing dogs, classifying colors, etc.). The resulting compound networks are jointly trained. We evaluate our approach on two challenging datasets for visual question answering, achieving state-of-the-art results on both the VQA natural image dataset and a new dataset of complex questions about abstract shapes.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Model Architectures

Citations in the corpus

Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, background): “multimodal fusion methods [14, 80, 27, 26]; (iii) multi-step reasoning [77, 22, 15]; (iv) incorporation of object relations [55, 48, 6, 32]; and (v) neural module networks for compositional reasoning [3, 25, 20, 11]. In principle, our proposed VILLA framework can be plugged into these “shallower” mo”
Environmental drivers of systematicity and generalization in a situated agent (2019, background): “, 2018), modular-components defined by symbolic parses (Andreas et al., 2016; Bahdanau et al., 2018), explicit latent variables (Higgins et al.”
Towards VQA Models That Can Read (2019, background): “VQA models typically use some variant of attention to get a representation of the image that is relevant for answering the question [2, 7, 30, 47, 48, 51, 17].”
ShapeWorld - A new test methodology for multimodal language understanding (2017, methodology): “In the near future, we plan to also adapt the technique of multimodal compact bilinear pooling (Fukui et al., 2016), neural module networks (Andreas et al., 2016a,b) and potentially also relation networks (Raposo et al., 2017) to the ICA task, and upload implementations to the GitHub repository.”
Programmable Agents (2017, background): “Neural Module Networks [2, 1], which use modularity in visual question answering, have been particularly inﬂuential on our thinking.”
Learning to Understand Goal Specifications by Modelling Reward (2018, methodology): “The k -th layer of the convnet performs a computation The same procedure as described above for h NMN is used to produce the network outputs using the output h 5 of the 5 th layer of the convnet.”
Learning to Reason Deductively: Math Word Problem Solving as Complex Relation Extraction (2022, methodology): “We design differentiable modules (Andreas et al., 2016; Gupta et al., 2020) in our model (§3.2) to perform reasoning among the quantities.”
Grammar-Based Grounded Lexicon Learning (2022, methodology): “(2) TbD-Net [31] uses a pre-trained semantic parser to parse the question into a symbolic program, and executes the program with a neural module network [5].”
Measuring Association Between Labels and Free-Text Rationales (2020, background): “One way towards faithfulness is to introduce architectural modiﬁcations or constraints that produce rationales with desirable properties (Andreas et al., 2016; Schwartz et al., 2018; Jiang et al., 2019, inter alia ).”
A Multi-Type Multi-Span Network for Reading Comprehension that Requires Discrete Reasoning (2019, methodology): “Later, Andreas et al. (2016a) proposed dynamically choosing an optimal layout structure from a list of layout candidates that are produced by off-the-shelf parsers.”
Neural Logic Machines (2019, background): “Besides, modular networks (Andreas et al., 2016; 2017; Mascharka et al., 2018) are proposed for the reasoning over subsymbolic data such as images and natural language question answering.”
Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning (2022, background): “For example, such modules may be neural networks or hand-crafted functions designed to attend to a single object, or to compare the location or size of two inputs (Andreas et al., 2016; Yi et al., 2018).”
ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks (2019, background): “These challenges may be approachable by models that exploit hierarchy [8, 26], modularity [4, 16], and structured reasoning and planning [6].”
Unsupervised Question Decomposition for Question Answering (2020, neutral): “Examples include classical QA systems like SHRDLU (Winograd, 1972) and LUNAR (Woods et al., 1974), as well as neural Seq2Seq semantic parsers (Dong & Lapata, 2016) and neural module networks (Andreas et al., 2015; 2016).”
OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge (2019, background): “Modular networks [2, 15, 19] leverage the compositional nature of the language in deep neural networks.”
VizWiz Grand Challenge: Answering Visual Questions from Blind People (2018, methodology): “0: Real [17] MSCOCO [25] Crowd workers (AMT) Typed Visual7W [45] MSCOCO [25] Crowd workers (AMT) Typed CLEVR [19] Synthetic Shapes Automatically generated (templates) ——– SHAPES [6] Synthetic Shapes Automatically generated (templates) ——– Visual Genome [23] MSCOCO [25] & YFCC100M [35] Crowd workers ”
ViperGPT: Visual Inference via Python Execution for Reasoning (2023, background): “We take inspiration from Neural Module Networks [1, 15], who argue that complex vision tasks are fundamentally compositional and propose dividing them into atomic perceptual units.”
Visual Programming: Compositional visual reasoning without training (2022, methodology): “For the visual question answering (VQA) task, Neural Module Networks (NMN) [2,9,10,12] compose a question-speciﬁc, end-to-end trainable network from specialized, differentiable neural modules.”