Corpora Generation for Grammatical Error Correction

By Jared Lichtarge, Christopher Alberti, Shankar Kumar, Noam Shazeer, Niki Parmar, Simon Tong. 2019

146 citations recorded by the source corpus.

Grammatical Error Correction (GEC) has been recently modeled using the sequence-to-sequence framework. However, unlike sequence transduction problems such as machine translation, GEC suffers from the lack of plentiful parallel data. We describe two approaches for generating large parallel datasets for GEC using publicly available Wikipedia data. The first method extracts source-target pairs from Wikipedia edit histories with minimal filtration heuristics while the second method introduces noise into Wikipedia sentences via round-trip translation through bridge languages. Both strategies yield similar sized parallel corpora containing around 4B tokens. We employ an iterative decoding strategy that is tailored to the loosely supervised nature of our constructed corpora. We demonstrate that neural GEC models trained using either type of corpora give similar performance. Fine-tuning these models on the Lang-8 corpus and ensembling allows us to surpass the state of the art on both the CoNLL ‘14 benchmark and the JFLEG task. We present systematic analysis that compares the two approaches to data generation and highlights the effectiveness of ensembling.

Subfields: Artificial Intelligence, Natural Language Processing, Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Natural Language Generation

Citations in the corpus

Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding (2021, background): “2019; Lichtarge et al., 2019; Zhou et al., 2020; Wan et al., 2020), most methods that improve performance (Ge et al., 2018b; Kaneko et al., 2020) introduce additional computational cost and thus slow down inference despite the performance improvement.”
Seq2Edits: Sequence Transduction Using Span-level Edit Operations (2020, methodology): “Our pre-trained models for all tasks are trained for 1M iterations on 170M sentences extracted from English Wikipedia revisions and 176M sentences from English Wikipedia round-trip translated via German (Lichtarge et al., 2019).”
Quiz-Style Question Generation for News Stories (2021, unknown): “mar error correction model [28] in order to fix minor grammar, spelling, punctuation, and capitalization errors.”
Encoder-Decoder Models Can Benefit from Pre-trained Masked Language Models in Grammatical Error Correction (2020, methodology): “This pseudo-data is generated by probabilistically injecting character errors into the output (Lichtarge et al., 2019) of a back-”
A Comprehensive Survey of Grammar Error Correction (2020, methodology): “Round -trip translation [107] [84] and 0.98 ERRANT 𝐹 0.5 [60]).”
Improving the Efficiency of Grammatical Error Correction with Erroneous Span Detection and Correction (2020, unknown): “…not only take time to rescore but also slow down the correction model with a larger beam size during inference; multi-round (iterative) decoding (Ge et al., 2018a,b; Lichtarge et al., 2019) needs to repeatedly run the model; BERT-fuse (Kaneko et al., 2020) adds extra computation for model fusion.”
Neural Grammatical Error Correction Systems with Unsupervised Pre-training on Synthetic Data (2019, methodology): “Junczys-Dowmunt et al. (2018b) utilized a large amount of monolingual data by pre-training decoder parameters with a language model, and Lichtarge et al. (2018, 2019), on the other hand, used a large-scale out-of-domain parallel corpus extracted from Wikipedia revisions to pre-train their models.”
Parallel Iterative Edit Models for Local Sequence Transduction (2019, methodology): “Owing to the remarkable success of this model in challenging tasks like translation, almost all state-of-the-art neural models for GEC use it (Zhao et al., 2019; Lichtarge et al., 2019; Ge et al., 2018b; Chol-lampatt and Ng, 2018b; Junczys-Dowmunt et al., 2018).”
Improving Grammatical Error Correction with Machine Translation Pairs (2019, methodology): “Data Generation from Round-trip Translations Round-trip translation (Lichtarge et al., 2019) is an alternative approach to synthesis GEC data.”
A Neural Grammatical Error Correction System Built On Better Pre-training and Sequential Transfer Learning (2019, methodology): “…Transformers pre-trained on random error denoising (Zhao et al., 2019), Transformers pre-trained on Wikipedia revisions and roundtrip translations (Lichtarge et al., 2019), hybrid statistical and neural machine translation systems (Junczys-Dowmunt et al., 2018), and convolutional seq2seq models…”
Controllable data synthesis method for grammatical error correction (2019, methodology): “Calculate the geometric average (Lichtarge et al., 2019) of all single model outputs as the final output of the ensemble model.”
Sequence-to-sequence Pre-training with Data Augmentation for Sentence Rewriting (2019, methodology): “In JFLEG, it achieves 63.54 GLEU score which is the new state-of-the-art result, even out-performing the multi-round decoding results of Grundkiewicz and Junczys-Dowmunt (2018) and Lichtarge et al. (2019).”
Out of Style: RAG's Fragility to Linguistic Variation (2025, unknown): “We alter the grammatical correctness through two approaches, inspired by Yin et al. (2020); Zhuang and Zuccon (2021); Lichtarge et al. (2019): (1) Typos , where random addition, deletion, or substitution operations are applied at a 20% probability per word, requiring an edit distance of at least 1;…”