Compositional generalization in a deep seq2seq model by separating syntax and semantics

By Jacob Russin, Jason Jo, Randall C. O’Reilly, Yoshua Bengio. 2019

105 citations recorded by the source corpus.

Standard methods in deep learning for natural language processing fail to capture the compositional structure of human language that allows for systematic generalization outside of the training distribution. However, human learners readily generalize in this way, e.g. by applying known grammatical rules to novel words. Inspired by work in neuroscience suggesting separate brain systems for syntactic and semantic processing, we implement a modification to standard approaches in neural machine translation, imposing an analogous separation. The novel model, which we call Syntactic Attention, substantially outperforms standard methods in deep learning on the SCAN dataset, a compositional generalization task, without any hand-engineered features or additional supervision. Our work suggests that separating syntactic from semantic learning may be a useful heuristic for capturing compositional structure.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Computational Linguistics & Semantics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Computational Linguistics & Semantics

Citations in the corpus

Good-Enough Compositional Data Augmentation (2019, neutral): “, 2015) and composition operators (Bowman et al., 2016; Russin et al., 2019).”
Measuring Compositional Generalization: A Comprehensive Method on Realistic Data (2019, methodology): “Russin et al. (2019) discuss a particular extension of a seq2seq model that is effective in handling difﬁcult SCAN sub-tasks by separating semantic and syntactic information during learning.”
Learning to Recombine and Resample Data for Compositional Generalization (2020, methodology): “(1) In addition to task-speciﬁc model architectures (Andreas et al., 2016; Russin et al., 2019), recent years have seen a renewed interest in data augmentation as a ﬂexible and model-agnostic tool for encouraging controlled generalization (Ratner et al., 2017).”
Unlocking Compositional Generalization in Pre-trained Models Using Intermediate Representations (2021, unknown): “These include new or modiﬁed model architectures (Li et al., 2019; Russin et al., 2019; Gordon et al., 2020; Liu et al., 2020; Nye et al., 2020; Chen et al., 2020; Zheng and Lapata, 2020; Oren et al., 2020; Herzig and Berant, 2020), hybrid models (Shaw et al., 2020), meta-learning (Lake, 2019), and…”
A Benchmark for Systematic Generalization in Grounded Language Understanding (2020, methodology): “nt manners of moving. Permutations in the instructions modify the actions in a non-local manner, in their terminology, a problem that would also affect the recent syntax-semantics separation approach [36]. Another new approach uses meta-learning to succeed on some SCAN splits [27]. This model ‘learn”
Compositional generalization through meta sequence-to-sequence learning (2019, methodology): “There has been recent progress on SCAN due to clever data augmentation [1] and syntax-based attention [30], although both approaches are currently limited in scope.”
Discovering the Compositional Structure of Vector Representations with Role Learning Networks (2019, result): “Recent work showed improvements in compositionality by separating out syntax and semantics with attention (Russin et al., 2019), and our results suggest that ROLE can also disentangle syntax and semantics.”
Compositional Generalization via Neural-Symbolic Stack Machines (2020, methodology): “To achieve generalization, the model needs to understand the relationship between “twice” and “thrice”, which is not captured by the primitive equivalence [25, 28, 40] or local equivariance [16] studied in prior work.”
Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both? (2020, methodology): “Among the works that do not need task-specific resources, we evaluate the two models with publicly available code: Syntactic Attention (Russin et al., 2019) and CGPS (Li et al.”
Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing (2022, unknown): “…2016; Andreas, 2020; Akyürek et al., 2021; Oren et al., 2021; Qiu et al., 2022; Yang et al., 2022), specialized architectures (Li et al., 2019; Russin et al., 2019; Gordon et al., 2020; Liu et al., 2020; Nye et al., 2020; Chen et al., 2020; Zheng and Lapata, 2021; Oren et al., 2020; Herzig…”
Sequence-to-Sequence Learning with Latent Neural Grammars (2021, unknown): “…has since been a large body of work on compositional sequence-to-sequence learning through various approaches including modiﬁcations to existing architectures [70, 94, 44, 17, 26], grammars and neuro-symbolic models [86, 98, 85, 19, 76], meta-learning [67, 25], and data augmentation [6, 49, 50, 4].”
Learning Compositional Rules via Neural Program Synthesis (2020, methodology): “Training Setup: Previous work on SCAN has used a variety of techniques, including data augmentation (Andreas, 2019), meta-learning (Lake, 2019), and syntactic attention (Russin et al., 2019).”