Learning to Recombine and Resample Data for Compositional Generalization

By Ekin Akyürek, Afra Feyza Akyürek, Jacob Andreas. 2020

82 citations recorded by the source corpus.

Flexible neural models outperform grammar- and automaton-based counterparts on a variety of sequence modeling tasks. However, neural models perform poorly in settings requiring compositional generalization beyond the training data -- particularly to rare or unseen subsequences. Past work has found symbolic scaffolding (e.g. grammars or automata) essential in these settings. Here we present a family of learned data augmentation schemes that support a large category of compositional generalizations without appeal to latent symbolic structure. Our approach to data augmentation has two components: recombination of original training examples via a prototype-based generative model and resampling of generated examples to encourage extrapolation. Training an ordinary neural sequence model on a dataset augmented with recombined and resampled examples significantly improves generalization in two language processing problems---instruction following (SCAN) and morphological analysis (Sigmorphon 2018)---where our approach enables learning of new constructions and tenses from as few as eight initial examples.

Subfields: Artificial Intelligence, Mechanical Engineering, Molecular Biology, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Mechanical Engineering

In Molecular Biology

In Representation & Self-Supervised Learning

Citations in the corpus

Grammar-Based Grounded Lexicon Learning (2022, methodology): “There are two representative approaches: building structured neural networks with explicit phrase-based structures or segments [37, 51, 41, 34]; and using data augmentation techniques [3, 17, 2].”
Least-to-Most Prompting Enables Complex Reasoning in Large Language Models (2022, background): “Another line of work on SCAN designs data augmentation schemes (Andreas, 2020; Akyürek et al., 2021; Lake, 2019).”
Tailor: Generating and Perturbing Text with Semantic Controls (2021, background): “ing existing textual fragments, within or across inputs (Akyürek et al., 2020; Andreas, 2020).”
Neural Data Augmentation via Example Extrapolation (2021, background): “There is a large body of research on data augmentation (Jia and Liang, 2016; Andreas, 2020; Akyürek et al., 2021, inter alia).”
Inductive biases for deep learning of higher-level cognition (2020, background): “However, a form of compositionality which we propose here that should be better incorporated in deep learning is the form called systematicity [145] defined by linguists, and more recently systematic generalization in machine learning papers [22,146,147].”
SeqZero: Few-shot Compositional Semantic Parsing with Sequential Prompts and Zero-shot Models (2022, methodology): “, 2020; Akyürek et al., 2020). Pretrained models has also been shown useful for compositional semantic parsing (Oren et al., 2020; Furrer et al., 2020). None of prior work used sequential prompts or zero-shot models for compositional generalization. Yang et al. (2022) adopted attention biases to all”
Compositional Semantic Parsing with Large Language Models (2022, background): “, 2020; Herzig & Berant, 2021) and training algorithms (Lake, 2019; Kim, 2021), training data augmentation (Andreas, 2020; Akyürek et al., 2021), and prompting (Zhou et al.”
Contrastive Distillation Is a Sample-Efficient Self-Supervised Loss Policy for Transfer Learning (2022, unknown): “…encourages compositionality across multiple retrievals (information bottleneck constraint) 5. the latent variables z t activated by u t are activated regardless of the location of u t in the solver’s input, which helps to prevent overﬁtting (Akyürek et al., 2021) (position-invariance constraint) 6.”
Sequence-to-Sequence Learning with Latent Neural Grammars (2021, unknown): “…has since been a large body of work on compositional sequence-to-sequence learning through various approaches including modiﬁcations to existing architectures [70, 94, 44, 17, 26], grammars and neuro-symbolic models [86, 98, 85, 19, 76], meta-learning [67, 25], and data augmentation [6, 49, 50, 4].”
How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Composition (2022, methodology): “A distinct approach by Akyürek et al. (2021) uses data augmentation to specifically target compositionality, combining prototypes of a generative model into multiprototype samples.”
Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing (2022, background): “…proposed to improve compositional generalization in semantic parsing, including compositional data augmentation (Jia and Liang, 2016; Andreas, 2020; Akyürek et al., 2021; Oren et al., 2021; Qiu et al., 2022; Yang et al., 2022), specialized architectures (Li et al., 2019; Russin et al., 2019;…”