Compositional Generalization for Primitive Substitutions

By Yuanpeng Li, Liang Zhao, Joel Hestness. 2019

89 citations recorded by the source corpus.

Compositional generalization is a basic mechanism in human language learning, but current neural networks lack such ability. In this paper, we conduct fundamental research for encoding compositionality in neural networks. Conventional methods use a single representation for the input sentence, making it hard to apply prior knowledge of compositionality. In contrast, our approach leverages such knowledge with two representations, one generating attention maps, and the other mapping attended input words to output symbols. We reduce the entropy in each representation to improve generalization. Our experiments demonstrate significant improvements over the conventional methods in five NLP tasks including instruction learning and machine translation. In the SCAN domain, it boosts accuracies from 14.0% to 98.8% in Jump task, and from 92.0% to 99.7% in TurnLeft task. It also beats human performance on a few-shot learning task. We hope the proposed approach can help ease future research towards human-level compositional language learning.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Least-to-Most Prompting Enables Complex Reasoning in Large Language Models (2022, background): “Other prior works propose neural network architectures to improve compositional generalization, where they encourage the model to learn the word and span mapping (Russin et al., 2019; Li et al., 2019), the alignment of input and output as span trees (Herzig & Berant, 2021), and the permutation equiv”
Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both? (2020, methodology): “This has motivated many specialized architectures with stronger compositional biases that improve peformance on SCAN (Li et al., 2019; Russin et al., 2019; Gordon et al., 2019; Lake, 2019; Liu et al., 2020; Nye et al., 2020; Chen et al., 2020).”
Compositional Generalization via Neural-Symbolic Stack Machines (2020, methodology): “We present the results in Table 4, where we compare NeSS with the standard sequence-to-sequence model [24], and the primitive substitution approach [28] discussed in Section 4.1.”
Leap-Of-Thought: Teaching Pre-Trained Models to Systematically Reason Over Implicit Knowledge (2020, background): “There have been numerous earlier demonstrations that neural systems can learn systematic behavior, including for semantic parsing [31], symbolic integration [32], mathematics [33], knowledge prediction [34, 35], and rule-based reasoning [4].”
Teaching Pre-Trained Models to Systematically Reason Over Implicit Knowledge (2020, background): “There have been numerous earlier demonstrations that neural systems can learn systematic behavior, including for semantic parsing [38], symbolic integration [39], mathematics [40], and rule-based reasoning [4].”
Sequence-to-Sequence Learning with Latent Neural Grammars (2021, unknown): “Our baselines here include standard LSTM/Transformer models as well as approaches that explicitly target compositional generalization [70, 19].”
Unlocking Compositional Generalization in Pre-trained Models Using Intermediate Representations (2021, background): “Both new architectures (Li et al., 2019; Lake, 2019; Nye et al., 2020; Chen et al., 2020, inter alia ) and general-purpose pre-trained seq2seq models such as T5 (Raffel et al., 2020) have shown improvements on some evaluations of compositional generalization, but strong performance in general…”
Compositional Continual Language Learning (2020, unknown): “Baselines include Compositional (Li et al., 2019), EWC (Kirkpatrick et al.”
Compositional Semantic Parsing with Large Language Models (2022, background): “, 2018), including specialized design of neural model architectures (Chen et al., 2020; Nye et al., 2020; Liu et al., 2020; Shaw et al., 2021; Russin et al., 2019; Li et al., 2019; Gordon et al., 2020; Herzig & Berant, 2021) and training algorithms (Lake, 2019; Kim, 2021), training data augmentation”
COGS: A Compositional Generalization Challenge Based on Semantic Interpretation (2020, background): “One illustrative example is the poor performance of LSTMs on a SCAN split that requires generalizing from shorter to longer sequences.”
Compositional Generalization by Learning Analytical Expressions (2020, background): “…two groups: (i) No Extra Resources includes vanilla sequence to sequence with attention (Seq2Seq) [20, 24], convolutional sequence to sequence (CNN) [10], Transformer [37], Universal Transformer [9], Syntactic Attention [30] and Compositional Generalization for Primitive Substitutions (CGPS) [22].”
Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing (2022, background): “…(Jia and Liang, 2016; Andreas, 2020; Akyürek et al., 2021; Oren et al., 2021; Qiu et al., 2022; Yang et al., 2022), specialized architectures (Li et al., 2019; Russin et al., 2019; Gordon et al., 2020; Liu et al., 2020; Nye et al., 2020; Chen et al., 2020; Zheng and Lapata, 2021; Oren et…”