Noising and Denoising Natural Language: Diverse Backtranslation for Grammar Correction

By Ziang Xie, Guillaume Genthial, Stanley Xie, Andrew Y. Ng, Dan Jurafsky. 2018

122 citations recorded by the source corpus.

Translation-based methods for grammar correction that directly map noisy, ungrammatical text to their clean counterparts are able to correct a broad range of errors; however, such techniques are bottlenecked by the need for a large parallel corpus of noisy and clean sentence pairs. In this paper, we consider synthesizing parallel data by noising a clean monolingual corpus. While most previous approaches introduce perturbations using features computed from local context windows, we instead develop error generation processes using a neural sequence transduction model trained to translate clean examples to their noisy counterparts. Given a corpus of clean examples, we propose beam search noising procedures to synthesize additional noisy examples that human evaluators were nearly unable to discriminate from nonsynthesized examples. Surprisingly, when trained on additional data synthesized using our best-performing noising scheme, our model approaches the same performance as when trained on additional nonsynthesized data.

Subfields: Artificial Intelligence, Natural Language Processing, Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Natural Language Generation

Citations in the corpus

Improving Grammatical Error Correction with Machine Translation Pairs (2019, methodology): “Prior studies (Rei et al., 2017; Xie et al., 2018; Ge et al., 2018; Zhao et al., 2019; Kiyono et al., 2019)”
Improving Grammatical Error Correction with Data Augmentation by Editing Latent Representation (2020, methodology): “Inspired by back-translation procedure for machine translation (Sennrich et al., 2015) , Xie et al.(2018) proposed a model that can learn to generate erroneous sentences from correct ones.”
Improving the Efficiency of Grammatical Error Correction with Erroneous Span Detection and Correction (2020, unknown): “However, except those adding synthetic erroneous data (Xie et al., 2018; Ge et al., 2018a; Grundkiewicz et al., 2019; Kiyono et al., 2019; Zhou et al., 2019) and Wikipedia revision logs (Lichtarge et al., 2019) for training, most methods cause an increase in latency.”
A Comprehensive Survey of Grammar Error Correction (2020, background): “However, the traditional beam search in decoder would result far fewer errors in generated ungrammatical sentences than original noisy text [103].”
Substructure Substitution: Structured Data Augmentation for NLP (2021, background): “…(Sennrich et al., 2016), which has been demonstrated effective on not only machine translation, but also style-transfer (Prabhumoye et al., 2018; Zhang et al., 2020a), conditional text generation (Sobrevilla Cabezudo et al., 2019), and grammatical error correction (Xie et al., 2018).”
Neural Grammatical Error Correction Systems with Unsupervised Pre-training on Synthetic Data (2019, methodology): “, 2018a,b), synthesizing noisy sentences (Xie et al., 2018), or pre-training a neural network on a largescale but out-of-domain parallel corpus from Wikipedia (Lichtarge et al.”
Corpora Generation for Grammatical Error Correction (2019, methodology): “Back-translation (Sennrich et al., 2016a; Xie et al., 2018) addresses data sparsity by introducing noise into a clean corpus using a translation model trained in the clean to noisy direction.”
Sequence-to-sequence Pre-training with Data Augmentation for Sentence Rewriting (2019, background): “For sentence rewriting, an important research branch for data augmentation is artificial error generation for GEC (Brockett et al., 2006; Foster and Andersen, 2009; Rozovskaya and Roth, 2010, 2011; Rozovskaya et al., 2012; Felice et al., 2014; Yuan et al., 2016; Rei et al., 2017; Xie et al., 2018), ”
Encoder-Decoder Models Can Benefit from Pre-trained Masked Language Models in Grammatical Error Correction (2020, methodology): “This pseudo-data is generated by probabilistically injecting character errors into the output (Lichtarge et al., 2019) of a back-translation (Xie et al., 2018) model that generates grammatically incorrect sentences from grammati-cally correct sentences (Kiyono et al., 2019).”
Controllable data synthesis method for grammatical error correction (2019, methodology): “Some works use rule-based methods to generate artificial error data that contains specific types of errors (Yuan and Felice, 2013; Xie et al., 2016; Rei et al., 2017).”
A Comprehensive Survey of Grammatical Error Correction (2021, background): “However, the traditional beam search in decoder would result in far fewer errors in generated ungrammatical sentences than original noisy texts [140].”
A Neural Grammatical Error Correction System Built On Better Pre-training and Sequential Transfer Learning (2019, methodology): “For example, in GEC, pre-trained DAEs have been used for correcting erroneous sentences (Xie et al., 2018; Zhao et al., 2019).”