Encoder-Decoder Models Can Benefit from Pre-trained Masked Language Models in Grammatical Error Correction

By Masahiro Kaneko, Masato Mita, Shun Kiyono, Jun Suzuki, Kentaro Inui. 2020

163 citations recorded by the source corpus.

This paper investigates how to effectively incorporate a pre-trained masked language model (MLM), such as BERT, into an encoder-decoder (EncDec) model for grammatical error correction (GEC). The answer to this question is not as straightforward as one might expect because the previous common methods for incorporating a MLM into an EncDec model have potential drawbacks when applied to GEC. For example, the distribution of the inputs to a GEC model can be considerably different (erroneous, clumsy, etc.) from that of the corpora used for pre-training MLMs; however, this issue is not addressed in the previous methods. Our experiments show that our proposed method, where we first fine-tune a MLM with a given GEC corpus and then use the output of the fine-tuned MLM as additional features in the GEC model, maximizes the benefit of the MLM. The best-performing model achieves state-of-the-art performances on the BEA-2019 and CoNLL-2014 benchmarks. Our code is publicly available at: https://github.com/kanekomasahiro/bert-gec.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Furu Wei
2. Tao Ge
3. Xin Sun
4. Xiaojun Wan
5. Houfeng Wang
6. Zhuowei Liu
7. Wenguang Wang
8. Zhaohong Wan

In Natural Language Processing

1. Furu Wei
2. Tao Ge
3. Neel Sundaresan
4. Xin Sun
5. Houfeng Wang
6. Zhuowei Liu
7. Alexey Svyatkovskiy
8. Xiaojun Wan

Citations in the corpus

• Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding (2021, background): “2019; Lichtarge et al., 2019; Zhou et al., 2020; Wan et al., 2020), most methods that improve performance (Ge et al., 2018b; Kaneko et al., 2020) introduce additional computational cost and thus slow down inference despite the performance improvement.”
• Improving the Efficiency of Grammatical Error Correction with Erroneous Span Detection and Correction (2020, unknown): “For example, language model and right-to-left (R2L) rescoring (Grundkiewicz et al., 2019; Kiyono et al., 2019) not only take time to rescore but also slow down the correction model with a larger beam size during inference; multi-round (iterative) decoding (Ge et al., 2018a,b; Lichtarge et al., 2019)”
• Improving Grammatical Error Correction with Data Augmentation by Editing Latent Representation (2020, methodology): “, 2019), can be incorporated into the encoder-decoder model (Kaneko et al., 2020).”
• Generating bug-fixes using pretrained transformers (2021, methodology): “Kaneko et al. use a seq2seq model where the input embeddings are concatenated with the embeddings produced by a BERT finetuned on grammatical error detection (detecting whether a given token is faulty) (Kaneko et al., 2020).”
• A Comprehensive Survey of Grammatical Error Correction (2021, methodology): “Inspired by the application of right-to-left model in machine translation [118], some GEC systems rerank hypothesis using the scores of right-to-left model to better incorporate the context of each word [51, 70, 74].”