Rearranging the Familiar: Testing Compositional Generalization in Recurrent Networks

By João Loula, Marco Baroni, Brenden M. Lake. 2018

137 citations recorded by the source corpus.

Systematic compositionality is the ability to recombine meaningful units with regular and predictable outcomes, and it’s seen as key to the human capacity for generalization in language. Recent work (Lake and Baroni, 2018) has studied systematic compositionality in modern seq2seq models using generalization to novel navigation instructions in a grounded environment as a probing tool. Lake and Baroni’s main experiment required the models to quickly bootstrap the meaning of new words. We extend this framework here to settings where the model needs only to recombine well-trained functional words (such as “around” and “right”) in novel contexts. Our findings confirm and strengthen the earlier ones: seq2seq models can be impressively good at generalizing to novel combinations of previously-seen input, but only when they receive extensive training on the specific pattern to be generalized (e.g., generalizing from many examples of “X around right” to “jump around right”), while failing when generalization requires novel application of compositional rules (e.g., inferring the meaning of “around right” from those of “right” and “around”).

Subfields: Artificial Intelligence, Molecular Biology, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

In Natural Language Processing

In Model Architectures

Citations in the corpus

On the Binding Problem in Artificial Neural Networks (2020, background): “For example, it is difficult for neural language models to generalize syntactic rules such as verb tenses or embedded clauses in a systematic manner (Keysers et al., 2020; Lake and Baroni, 2018; Loula et al., 2018; Hupkes et al., 2020).”
Measuring Compositional Generalization: A Comprehensive Method on Realistic Data (2019, background): “Lake & Baroni (2018) introduce the SCAN dataset, and several publications provide interesting analyses of compositional generalization using it (Bastings et al., 2018; Loula et al., 2018).”
Compositional Generalization via Neural-Symbolic Stack Machines (2020, background): “In contrast, deep neural networks, including the state-of-the-art models for natural language understanding, typically lack such generalization abilities [24, 22, 30], although they have demonstrated impressive performance on various applications.”
Compositional generalization in a deep seq2seq model by separating syntax and semantics (2019, background): “Recent work has shown that standard deep learning methods in natural language processing fail to capture this important property: when tested on unseen combinations of known elements, state-ofthe-art models fail to generalize [15; 17; 4].”
Learning to Recombine and Resample Data for Compositional Generalization (2020, background): “The around right split (Loula et al., 2018) tests zero-shot generalization by presenting learners with constructions like walk around left and walk right in the training set, but walk around right only in the test set.”
Transcoding Compositionally: Using Attention to Find More Generalizable Solutions (2019, methodology): “As Loula et al. (2018) observed a great difference in performance between the 0 ﬁllers and 1 ﬁller conditions, they zoom in on these conditions in experiment 3.”
A Benchmark for Systematic Generalization in Grounded Language Understanding (2020, background): “These experiments are related to the ‘around right’-split introduced in SCAN by Loula et al. (2018), but in this case, the grounded meaning of each adverb in the world state changes its eﬀect on the target sequence.”
Unlocking Compositional Generalization in Pre-trained Models Using Intermediate Representations (2021, methodology): “For SCAN, as no dev set is available, we tune the learning rate on the around right split (Loula et al., 2018), which we use as a held-out set.”
Human few-shot learning of compositional instructions (2019, background): “These types of inferences are critical to language learning and understanding, yet modern recurrent neural networks struggle to generalize in similarly systematic ways (Lake & Baroni, 2018; Loula et al., 2018).”
Compositional Generalization by Learning Analytical Expressions (2020, background): “In terms of the usage of extra resources, we divide them into two groups: (i) No Extra Resources includes vanilla sequence to sequence with attention (Seq2Seq) [19, 23], convolutional sequence to sequence (CNN) [10], Transformer [36], Universal Transformer [9], Syntactic Attention [29] and Compositi”
Case-based Reasoning for Natural Language Queries over Knowledge Bases (2021, background): “It has been shown that neural seq2seq models do not generalize well to novel combinations of previously seen input (Lake and Baroni, 2018; Loula et al., 2018).”
Compositional Generalization for Primitive Substitutions (2019, methodology): “The baseline methods are from Loula et al. (2018b).”
Learning Compositional Rules via Neural Program Synthesis (2020, methodology): “Despite advances, recent work shows that contemporary neural architectures still struggle to generalize in systematic ways when directly learning rule-like mappings between input sequences and output sequences [4, 5].”
The EOS Decision and Length Extrapolation (2020, background): “3 Specifically, this extra difﬁculty stems from the fact that neural models only perform the SCAN task well when they are trained with sufﬁcient examples of certain input sequence templates (Loula et al., 2018).”
CNNs found to jump around more skillfully than RNNs: Compositional Generalization in Seq2seq Convolutional Networks (2019, background): “Lake and Baroni (2018) and Loula et al. (2018) proposed the SCAN challenge to directly assess the ability of sequence-to-sequence networks to perform systematic, compositional generalization of linguistic rules.”