Compositional generalization through meta sequence-to-sequence learning

By Brenden M. Lake. 2019

206 citations recorded by the source corpus.

People can learn a new concept and use it compositionally, understanding how to "blicket twice" after learning how to "blicket." In contrast, powerful sequence-to-sequence (seq2seq) neural networks fail such tests of compositionality, especially when composing new concepts together with existing concepts. In this paper, I show how memory-augmented neural networks can be trained to generalize compositionally through meta seq2seq learning. In this approach, models train on a series of seq2seq problems to acquire the compositional skills needed to solve new seq2seq problems. Meta se2seq learning solves several of the SCAN tests for compositional learning and can learn to apply implicit rules to variables.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Environmental drivers of systematicity and generalization in a situated agent (2019, background): “On the other hand, some recent work has reported instances of strong generalisation in the absence of such speciﬁc inductive biases (Chaplot et al., 2018; Yu et al., 2018; Lake, 2019).”
Least-to-Most Prompting Enables Complex Reasoning in Large Language Models (2022, background): “Another line of work on SCAN designs data augmentation schemes (Andreas, 2020; Akyürek et al., 2021; Lake, 2019).”
Learning Compositional Rules via Neural Program Synthesis (2020, methodology): “Training Setup: Previous work on SCAN has used a variety of techniques [14, 9, 15].”
The EOS Decision and Length Extrapolation (2020, methodology): “In NLP, while better modeling techniques have improved models’ abilities to perform some kinds of systematic generalization (Gordon et al., 2019; Lake, 2019), these models still struggle to extrapolate; they have trouble producing and processing sequences longer than those seen during training even…”
Compositional Generalization via Neural-Symbolic Stack Machines (2020, methodology): “We observe that, when two sequences only differ in the arguments of REDUCE , their corresponding tokens could be mapped to the same category, which is the main focus of most prior work on compositional generalization [16, 28, 25].”
Compositional Generalization by Learning Analytical Expressions (2020, methodology): “(ii) Using Extra Resources consists of Good Enough Compositional Data Augmentation (GECA) [2], meta sequence to sequence (Meta Seq2seq) [19], equivariant sequence to sequence (Equivariant Seq2seq) [14] and program synthesis [28].”
LTL2Action: Generalizing LTL Instructions for Multi-Task RL (2021, background): “We note that some existing works have tackled this setting (Hill et al., 2021; Leon et al., 2020; Lake, 2019).”
Learning to Learn Words from Visual Scenes (2019, background): “However, unlike [24], our approach works for realistic language and natural images.”
Few-shot Sequence Learning with Transformers (2020, background): “Most prior work on few-shot learning have focused on computer vision benchmarks such as Omniglot (Lake, 2019) and Mini-ImageNet (Vinyals et al., 2016).”
COGS: A Compositional Generalization Challenge Based on Semantic Interpretation (2020, background): “While several models have made significant improvements over other SCAN splits, progress on the length split remains minimal (Li et al., 2019; Lake, 2019; Gordon et al., 2020).”
Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing (2022, background): “…variations (Csordás et al., 2021; Ontanón et al., 2021), intermediate representations (Herzig et al., 2021; Shin et al., 2021), meta-learning (Lake, 2019; Conklin et al., 2021; Zhu et al., 2021), and auxiliary objectives to bias attention in encoder-decoder models (Yin et al., 2021; Jiang…”
Word meaning in minds and machines (2020, background): “Other work has explored meta-learning as means of encouraging generalization to new situations and novel combinations of instructions (Lake, 2019).”
Compositional Semantic Parsing with Large Language Models (2022, background): “Prior efforts to improve compositional generalization primarily rely on specialized architectures or training procedures (Lake, 2019; Chen et al., 2020; Nye et al., 2020; Andreas, 2020; Conklin et al., 2021; Akyürek et al., 2021; Liu et al., 2021).”
Emergent Systematic Generalization in a Situated Agent (2019, background): “With respect to generalization, systematicity and neural networks, our ﬁndings contrast with other recent contributions (Lake, 2019; Bahdanau et al., 2018), and are more aligned with older empirical analyses of neural networks (McClelland et al., 1987; Frank, 2006).”
Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both? (2020, methodology): “This has motivated many specialized architectures with stronger compositional biases that improve peformance on SCAN (Li et al., 2019; Russin et al., 2019; Gordon et al., 2019; Lake, 2019; Liu et al., 2020; Nye et al., 2020; Chen et al., 2020).”
Few-Shot Learning With a Strong Teacher (2021, background): “…in artiﬁcial intelligence towards few-shot learning (FSL) 1 , especially in the applications of visual recognition [3], [4], machine translation [5], [6], reinforcement learning [7], [8], etc. Speciﬁcally for visual recognition, FSL is generally studied in a two-stage setting [9], [10], [11].”