Neural Grammatical Error Correction Systems with Unsupervised Pre-training on Synthetic Data
By Roman Grundkiewicz, Marcin Junczys-Dowmunt, Kenneth Heafield. 2019
191 citations recorded by the source corpus.
Considerable effort has been made to address the data sparsity problem in neural grammatical error correction. In this work, we propose a simple and surprisingly effective unsupervised synthetic error generation method based on confusion sets extracted from a spellchecker to increase the amount of training data. Synthetic data is used to pre-train a Transformer sequence-to-sequence model, which not only improves over a strong baseline trained on authentic error-annotated data, but also enables the development of a practical GEC system in a scenario where little genuine error-annotated data is available. The developed systems placed first in the BEA19 shared task, achieving 69.47 and 64.24 F_{0.5} in the restricted and low-resource tracks respectively, both on the W&I+LOCNESS test set. On the popular CoNLL 2014 test set, we report state-of-the-art results of 64.16 M² for the submitted system, and 61.30 M² for the constrained system trained on the NUCLE and Lang-8 data.
Subfields: Artificial Intelligence, Natural Language Processing, Natural Language Generation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Natural Language Processing
In Natural Language Generation
Citations in the corpus
- A Survey of Data Augmentation Approaches for NLP (2021, neutral): “, 2020) × × tok Input 3 UEDIN-MS (DA part) (Grundkiewicz et al., 2019) 3 × tok Input 3 NONCE (Gulordava et al.”
- Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding (2021, methodology): “To compare with the state-of-the-art approaches in English GEC that pretrain with synthetic data, we also synthesize 300M error-corrected sentence pairs for pretraining the English GEC model following the approaches of Grundkiewicz et al. (2019) and Zhang et al. (2019).”
- Encode, Tag, Realize: High-Precision Text Editing (2019, methodology): “…virtually every text generation task, including summarization (Tan et al., 2017), image captioning (Xu et al., 2015), text style transfer (Rao and Tetreault, 2018; Nikolov and Hahnloser, 2018; Jin et al., 2019), and grammatical error correction (Chollampatt and Ng, 2018; Grundkiewicz et al., 2019).”
- A Comprehensive Survey of Grammatical Error Correction (2021, methodology): “Inspired by the application of right-to-left model in machine translation [118], some GEC systems rerank hypothesis using the scores of right-to-left model to better incorporate the context of each word [51, 70, 74].”
- Seq2Edits: Sequence Transduction Using Span-level Edit Operations (2020, methodology): “This is very similar to the iterative decoding strategies used by Lichtarge et al. (2019); Awasthi et al. (2019); Ge et al. (2018); Grundkiewicz et al. (2019) with the difference that we pass through n-best lists between beam search passes rather than only the single best hypothesis.”
- A Comparative Study of Synthetic Data Generation Methods for Grammatical Error Correction (2020, background): “Training specifics The UEDIN-MS system generated parallel artificial data by applying the In-verted Spellchecker method to 100 million sentences sampled from the WMT News Crawl corpus.”
- Encoder-Decoder Models Can Benefit from Pre-trained Masked Language Models in Grammatical Error Correction (2020, methodology): “Numerous studies on GEC have successfully used encoder-decoder (EncDec) based models, and in fact, most current state-of-the-art neural GEC models employ this architecture (Zhao et al., 2019; Grundkiewicz et al., 2019; Kiyono et al., 2019).”
- A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios (2020, methodology): “For instance, Grundkiewicz et al. (2019) reverse the augmentation setting by applying trans-”
- Lossless Acceleration for Seq2seq Generation with Aggressive Decoding (2022, methodology): “To compare with the state-of-the-art approaches in English GEC that pretrain with synthetic data, we also synthesize 300M error-corrected sentence pairs for pretraining the English GEC model following the approaches of Grundkiewicz et al. (2019) and Zhang et al. (2019).”
- Improving Grammatical Error Correction with Machine Translation Pairs (2019, methodology): “Method BEA-19 CoNLL-14 Junczys-Dowmunt et al. (2018) - 53.”
- The BEA-2019 Shared Task on Grammatical Error Correction (2019, methodology): “Shuyao was also particularly adept at correcting multi to-ken errors, coming second after UEDIN-MS over-all.”
- Sequence-to-sequence Pre-training with Data Augmentation for Sentence Rewriting (2019, background): “…seq2seq learning (Luong et al., 2015), there is no much work related to seq2seq pre-training with data augmentation until in the last few months when some work (Lichtarge et al., 2019; Zhao et al., 2019; Grundkiewicz et al., 2019) have started to explore pre-training with augmented data for GEC.”
- Improving the Efficiency of Grammatical Error Correction with Erroneous Span Detection and Correction (2020, unknown): “However, except those adding synthetic erroneous data (Xie et al., 2018; Ge et al., 2018a; Grundkiewicz et al., 2019; Kiyono et al., 2019; Zhou et al., 2019) and Wikipedia revision logs (Lichtarge et al., 2019) for training, most methods cause an increase in latency.”
- Gender-Aware Reinflection using Linguistically Enhanced Neural Models (2020, unknown): “Sequence-to-sequence models have achieved significant results in grammatical error correction (GEC) (Chollampatt and Ng, 2018; Junczys-Dowmunt et al., 2018; Grundkiewicz et al., 2019) and morphological reinflection tasks (Faruqui et al., 2016; Kann and Schütze, 2016; Aharoni and Goldberg, 2017).”