Compositional Attention Networks for Machine Reasoning

By Drew A. Hudson, Christopher D. Manning. 2018

624 citations recorded by the source corpus.

We present the MAC network, a novel fully differentiable neural network architecture, designed to facilitate explicit and expressive reasoning. MAC moves away from monolithic black-box neural architectures towards a design that encourages both transparency and versatility. The model approaches problems by decomposing them into a series of attention-based reasoning steps, each performed by a novel recurrent Memory, Attention, and Composition (MAC) cell that maintains a separation between control and memory. By stringing the cells together and imposing structural constraints that regulate their interaction, MAC effectively learns to perform iterative reasoning processes that are directly inferred from the data in an end-to-end approach. We demonstrate the model's strength, robustness and interpretability on the challenging CLEVR dataset for visual reasoning, achieving a new state-of-the-art 98.9% accuracy, halving the error rate of the previous best model. More importantly, we show that the model is computationally-efficient and data-efficient, in particular requiring 5x less data than existing models to achieve strong results.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, background): “Prominent work includes: (i) advanced attention mechanisms [40, 79, 51]; (ii) better multimodal fusion methods [14, 80, 27, 26]; (iii) multi-step reasoning [77, 22, 15]; (iv) incorporation of object relations [55, 48, 6, 32]; and (v) neural module networks for compositional reasoning [3, 25, 20, 11]”
Grammar-Based Grounded Lexicon Learning (2022, methodology): “(1) MAC [21] is an end-to-end approach based on attention.”
Explainable Multi-hop Verbal Reasoning Through Internal Monologue (2021, background): “Typical works include the Differential Neural Computer (DNC) (Graves et al., 2016), End-to-end Memory Networks (Sukhbaatar et al., 2015), Dynamic Memory Networks (DMN) (Kumar et al., 2016) and Compositional Attention Networks (MAC) (Hudson and Manning, 2018).”
ViperGPT: Visual Inference via Python Execution for Reasoning (2023, background): “[15] and other work in this direction [12, 14, 31] typically implement π with a neural network that is trained with either supervised or reinforcement learning in order to estimate programs from queries.”
GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering (2019, methodology): “across the board, with a host of datasets being constructed [4, 11, 15, 41, 20] and numerous models being proposed [5, 38, 6, 10, 12].”
A Corpus for Reasoning about Natural Language Grounded in Photographs (2018, methodology): “Finally, we evaluate the Memory, Attention, and Composition approach (MAC; Hudson and Manning, 2018), which uses a sequence of attention-based steps.”
Fusion of Detected Objects in Text for Visual Question Answering (2019, background): “MAC also reduces the textual information to a vector of ﬁxed length.”
Visual Entailment: A Novel Task for Fine-Grained Image Understanding (2019, methodology): “Recent work on compositional reasoning systems have achieved nearly 100% results on CLEVR [29] but the SOTA performance on VQA-v2.”
On the Binding Problem in Artificial Neural Networks (2020, methodology): “…GNNs by updating each memory location in parallel (Henaff et al., 2017; Kaiser and Sutskever, 2016) or incorporate specialized structure for reasoning into the processor, e.g. for the purpose of visual question answering using a read-only memory (knowledge base; see Hudson and Manning, 2018).”
CLEVRER: CoLlision Events for Video REpresentation and Reasoning (2019, background): “The compositional reasoning models that performs well on the descriptive questions (MAC (V) and TbD-net (V)) only achieve marginal improvements over the random and language-only baselines on the causal tasks.”
Relational inductive biases, deep learning, and graph networks (2018, methodology): “…methods and deep learning (e.g., Reed and De Freitas, 2016; Garnelo et al., 2016; Ritchie et al., 2016; Wu et al., 2017; Denil et al., 2017; Hudson and Manning, 2018), we see great promise in synthesizing new techniques by drawing on the full AI toolkit and marrying the best approaches…”
REMIND Your Neural Network to Prevent Catastrophic Forgetting (2019, methodology): “, attention-based [6,46,78], compositional [7,28], bi-modal fusion [10,19,71]) and it can be applied to similar tasks like image captioning [12] and referring expression recognition [43,63,67].”
The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision (2019, methodology): “The SOTA methods do not provide interpretable representation on individual objects (Johnson et al., 2017a; Hudson & Manning, 2018; Mascharka et al., 2018) .”
Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding (2018, methodology): “Among the baseline methods we compare with, MAC [Hudson and Manning, 2018] obtains high accuracy with zero program annotations; in comparison, our method needs to be pretrained on 270 program annotations, but requires fewer question-answer pairs to reach similar performance.”
GQA: a new dataset for compositional question answering over real-world images (2019, methodology): “Beside these, we evaluate the performance of the bottom-up attention model [4] – the winner of 2017 VQA challenge, and the MAC model [11] – a compositional attention state-of-the-art model for CLEVR [15].”
Measuring Compositional Generalization: A Comprehensive Method on Realistic Data (2019, background): “Hudson & Manning (2018) describe how the application of compositional attention networks to the CLEVR data leads to structured and data-efﬁcient learning.”
Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog (2019, background): “, 2018), Visual Question Answering (VQA) (Yang et al., 2016; Nam et al., 2017; Hudson and Manning, 2018), and Machine Reading Comprehension (MRC) (Cui et al.”