A Good Sample is Hard to Find: Noise Injection Sampling and Self-Training for Neural Language Generation Models

By Chris Kedzie, K. McKeown. 2019

37 citations recorded by the source corpus.

Deep neural networks (DNN) are quickly becoming the de facto standard modeling method for many natural language generation (NLG) tasks. In order for such models to truly be useful, they must be capable of correctly generating utterances for novel meaning representations (MRs) at test time. In practice, even sophisticated DNNs with various forms of semantic control frequently fail to generate utterances faithful to the input MR. In this paper, we propose an architecture agnostic self-training method to sample novel MR/text utterance pairs to augment the original training data. Remarkably, after training on the augmented data, even simple encoder-decoder models with greedy decoding are capable of generating semantically correct utterances that are as good as state-of-the-art outputs in both automatic and human evaluations of quality.

Subfields: Natural Language Processing, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Optimization & Training

Citations in the corpus

Structure-to-Text Generation with Self-Training, Acceptability Classifiers and Context-Conditioning for the GEM Shared Task (2021, methodology): “including earlier work on automatic data cleaning— make use of cycle consistency between parsing and generation models (Chisholm et al., 2017; Nie et al., 2019; Kedzie and McKeown, 2019; Qader et al., 2019).”
Improving Factual Consistency of Abstractive Summarization via Question Answering (2021, background): “Improving ROUGE is not the goal of our paper; what we show is that we can significantly improve factual consistency of summaries without degrading ROUGE, as is common practice (Kedzie and McKeown, 2019).”
Text Generation by Learning from Demonstrations (2020, unknown): “Recently, there is an increasing amount of work on discourag-12 ing implausible samples during training, e.g., using negative sampling (Welleck et al., 2020b), self-training on high-quality samples (Kedzie & McKeown, 2019), and conﬁdence-oriented decoding with calibration (Tian et al., 2020).”
AggGen: Ordering and Aggregating while Generating (2021, background): “The prevailing sequence-to-sequence (seq2seq) architectures typically address this issue via reranking (Wen et al., 2015a; Dušek and Jurčı́ček, 2016; Juraska et al., 2018) or some sophisticated training techniques (Nie et al., 2019; Kedzie and McKeown, 2019; Qader et al., 2019).”
Evaluating Semantic Accuracy of Data-to-Text Generation with Natural Language Inference (2020, background): “Faithful NLG Some recent neural NLG systems train specifically for semantic accuracy (Nie et al., 2019; Tian et al., 2019; Kedzie and McKeown, 2019).”
Sticking to the Facts: Confident Decoding for Faithful Data-to-Text Generation (2019, background): “…rewards (Paulus et al., 2018; Pasunuru & Bansal, 2018), semi-Markov models to learn neural templates (Wiseman et al., 2018), data augmentation (Ma et al., 2019; Kedzie & McKeown, 2019), content planning (Puduppully et al., 2019a), and constrained vocabulary decoding (Wu et al., 2017).”