Unnatural Language Processing: Bridging the Gap Between Synthetic and Natural Language Data

By Alana Marzoev, Samuel Madden, M. Frans Kaashoek, Michael J. Cafarella, Jacob Andreas. 2020

33 citations recorded by the source corpus.

Large, human-annotated datasets are central to the development of natural language processing models. Collecting these datasets can be the most challenging part of the development process. We address this problem by introducing a general purpose technique for ``simulation-to-real'' transfer in language understanding problems with a delimited set of target behaviors, making it possible to develop models that can interpret natural utterances without natural training data. We begin with a synthetic data generation procedure, and train a model that can accurately interpret utterances produced by the data generator. To generalize to natural utterances, we automatically find projections of natural language utterances onto the support of the synthetic language, using learned sentence embeddings to define a distance metric. With only synthetic training data, our approach matches or outperforms state-of-the-art models trained on natural language data in several domains. These results suggest that simulation-to-real transfer is a practical framework for developing NLP applications, and that improved models for transfer might provide wide-ranging improvements in downstream tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

LILA: Language-Informed Latent Actions (2021, methodology): “To do this, adding linguistically diverse data has been the gold standard [44, 51]; however, a new class of approaches have emerged that sidestep additional data requirements by tapping into the potential of pretrained language models [43, 50].”
ALFWorld: Aligning Text and Embodied Environments for Interactive Learning (2020, background): “While these preliminary results with natural language are encouraging, we expect future work could augment the templated language with synthetic-to-real transfer methods (Marzoev et al., 2020) for better generalization.”
Improving Coherence and Consistency in Neural Sequence Models with Dual-System, Neuro-Symbolic Reasoning (2021, background): “It also includes semantic parsing systems trained either without supervision (Liang et al., 2017; Mou et al., 2017; Muhlgay et al., 2019), or with synthetic language data (Marzoev et al., 2020; Xu et al., 2020b).”
AutoQA: From Databases to Q&A Semantic Parsers with Only Synthetic Training Data (2020, background): “They are either synthetic-only (Marzoev et al., 2020) or use human data from other Overnight domains (Herzig and Berant, 2018b).”
Effective Distant Supervision for Temporal Relation Extraction (2020, background): “This observation follows a trend of recent work showing pre-trained models’ ability to generalize from synthetic data to natural data (Xu et al., 2020; Marzoev et al., 2020).”
SeqZero: Few-shot Compositional Semantic Parsing with Sequential Prompts and Zero-shot Models (2022, methodology): “Wang et al. (2015); Marzoev et al. (2020) generated paraphrase candidate values from a grammar of legal canonical utterances, and incrementally ﬁltered the bottom-up or top-down generation by scoring the partial candidates against ﬁnal formal language.”
Constrained Language Models Yield Few-Shot Semantic Parsers (2021, methodology): “…excel at generating natu-ral language, we should formulate semantic parsing as paraphrasing into a controlled sublanguage (Be-rant and Liang, 2014; Marzoev et al., 2020) and (2) autoregressive LMs can be efﬁciently constrained to search over only valid paraphrases, so the sub-language does not…”