Generalization without Systematicity: On the Compositional Skills of Sequence-to-Sequence Recurrent Networks

By Brenden M. Lake, Marco Baroni. 2017

965 citations recorded by the source corpus.

Subfields: Natural Language Processing, Model Architectures, Computational Linguistics & Semantics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Model Architectures

In Computational Linguistics & Semantics

Citations in the corpus

Augmented Language Models: a Survey (2023, background): “This recursive approach slightly differs from CoT since the latter does not explicitly formulate sub-problems, and can be especially useful for complex tasks, given that compositional generalization can be challenging for LMs (Lake and Baroni, 2018; Keysers et al., 2019; Li et al., 2022a).”
Environmental drivers of systematicity and generalization in a situated agent (2019, background): “An additional limitation of our work is that – much like closely-related work (Lake & Baroni, 2017; Bahdanau et al., 2018) – we do not provide a nomological explanation for the generalisation (or lack thereof) that we observe.”
Grammar-Based Grounded Lexicon Learning (2022, methodology): “We evaluate G2L2 on two domains: visual reasoning in CLEVR [23] and language-driven navigation in SCAN [27].”
Least-to-Most Prompting Enables Complex Reasoning in Large Language Models (2022, background): “In the prompt constructed to solve SCAN, we actually used both natural language and Python notation.”
Enhancing the Transformer with Explicit Relational Encoding for Math Problem Solving (2019, background): “…to enable the fully distributed representation of symbolic structure (like a hierarchy of features) which has recently resurfaced as an important direction for neural network research (Lake & Baroni, 2017; Bahdanau et al., 2018; van Steenkiste et al., 2019; Palangi et al., 2017; Tang et al., 2018).”
Compositionality Decomposed: How do Neural Networks Generalise? (2019, background): “…it means to say that a model is or is not compositional, a point exempliﬁed by the vast number of different tests that exist for compositionality [ Lake and Baroni, 2018; Hupkes et al. , 2018; Johnson et al. , 2017; Bahdanau et al. , 2018; Saxton et al. , 2019; Loula et al. , 2018; Dess`ı and…”
Analysis Methods in Neural Language Processing: A Survey (2018, background): “A few targeted attacks include Liang et al. (2018), which specified a desired class to fool a text classifier, and Chen et al.”
CommonGen: A Constrained Text Generation Challenge for Generative Commonsense Reasoning (2020, background): “This encourages models to reason about a potentially inﬁnite number of novel combinations of familiar concepts – an ability believed to be a limitation of current AI systems (Lake and Baroni, 2017; Keysers et al., 2020).”
On the Binding Problem in Artificial Neural Networks (2020, background): “For example, it is difficult for neural language models to generalize syntactic rules such as verb tenses or embedded clauses in a systematic manner (Keysers et al., 2020; Lake and Baroni, 2018; Loula et al., 2018; Hupkes et al., 2020).”
Learning First-Order Symbolic Representations for Planning from the Structure of the State Space (2019, methodology): “Yet by not constructing ﬁrst-order representations, DRL methods lose the beneﬁts of transparency, reusability, and compositionality [27, 25].”
Continual Lifelong Learning in Natural Language Processing: A Survey (2020, background): “In particular, widely used in NLP sequence-to-sequence models still struggle with systematic generalization (Lake and Baroni, 2018; Bahdanau et al., 2019), being unable to learn general rules and reason about high-level language concepts.”
The MAGICAL Benchmark for Robust Imitation (2020, background): “There are also several instruction-following benchmarks for evaluating generalisation in natural language understanding [29, 34].”
The Consciousness Prior (2017, background): “This is important in order to obtain systematic generalization [Lake and Baroni, 2017] and combinatorial properties omnipresent in natural language, making it easier to combine different pieces of neural hardware together dynamically, with keys being used to decide which information should be…”
Evaluating Compositionality in Sentence Embeddings (2018, background): “The failure of neural network models to achieve compositionality has been a recurring (and controversial) theme in cognitive science (Fodor & Pylyshyn, 1988; Gershman & Tenenbaum, 2015; Lake & Baroni, 2017).”
Assessing Composition in Sentence Vector Representations (2018, background): “Lake and Baroni (2017) investigate the capacity of RNNs to perform zero-shot generalization using composition, and Dasgupta et al. (2018) construct an entailment dataset with balanced lexical content in order to target composition more effectively.”
Automatically Composing Representation Transformations as a Means for Generalization (2018, background): “structure in RNNs related to that discussed in [52].”
Good-Enough Compositional Data Augmentation (2019, methodology): “Here we use the SCAN dataset (Lake and Baroni, 2018), which consists of simple English commands like jump around right paired with sequences of discrete actions like RTURN JUMP RTURN JUMP RTURN JUMP RTURN JUMP .”
Measuring Compositional Generalization: A Comprehensive Method on Realistic Data (2019, methodology): “As Lake & Baroni (2018) put it: “Once a person learns the meaning of a new verb ‘dax’, he or she can immediately understand the meaning of ‘dax twice’ and ‘sing and dax’.””
Probing Natural Language Inference Models through Semantic Fragments (2019, background): “…even for humans, but that are nonetheless based on sound formal principles (e.g., reasoning about monotonicity where, in contrast to the simple example in Figure 1, several nested downward monotone contexts are involved to test the model’s capacity for compositionality, cf. Lake and Baroni (2017)).”
WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020, neutral): “Outside of visual object recognition, other work has used synthetic datasets and transformations to explore compositional generalization, e.g., SCAN (Lake and Baroni, 2018).”