Fluency Boost Learning and Inference for Neural Grammatical Error Correction

By Tao Ge, Furu Wei, M. Zhou. 2018

107 citations recorded by the source corpus.

Most of the neural sequence-to-sequence (seq2seq) models for grammatical error correction (GEC) have two limitations: (1) a seq2seq model may not be well generalized with only limited error-corrected data; (2) a seq2seq model may fail to completely correct a sentence with multiple errors through normal seq2seq inference. We attempt to address these limitations by proposing a fluency boost learning and inference mechanism. Fluency boosting learning generates fluency-boost sentence pairs during training, enabling the error correction model to learn how to improve a sentence’s fluency from more instances, while fluency boosting inference allows the model to correct a sentence incrementally with multiple inference steps until the sentence’s fluency stops increasing. Experiments show our approaches improve the performance of seq2seq models for GEC, achieving state-of-the-art results on both CoNLL-2014 and JFLEG benchmark datasets.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding (2021, background): “2019; Lichtarge et al., 2019; Zhou et al., 2020; Wan et al., 2020), most methods that improve performance (Ge et al., 2018b; Kaneko et al., 2020) introduce additional computational cost and thus slow down inference despite the performance improvement.”
The BEA-2019 Shared Task on Grammatical Error Correction (2019, methodology): “…have been trained, tuned and tested on different combinations of metrics and corpora (Sakaguchi et al., 2017; Yannakoudakis et al., 2017; Chollampatt and Ng, 2018a; Ge et al., 2018; Grundkiewicz and Junczys-Dowmunt, 2018; Junczys-Dowmunt et al., 2018; Lichtarge et al., 2018; Zhao et al., 2019).”
Improving Grammatical Error Correction with Machine Translation Pairs (2019, methodology): “2017), neural machine translation (NMT) based GEC models (Chollampatt and Ng 2018; Junczys-Dowmunt et al. 2018; Ge et al. 2018) achieved state-of-the-art performance on this task.”
Erroneous data generation for Grammatical Error Correction (2019, background): “Later, various pure neu-Corpus ral systems (Ge et al., 2018; Junczys-Dowmunt et al., 2018; Lichtarge et al., 2018) reported state-of-the-art results successively.”
A Comprehensive Survey of Grammar Error Correction (2020, methodology): “Many NMT based GEC systems utilize a spellchecker 1 or open-source spellchecking library in preprocessing [68], [50], [53], [67], [66], [39].”
A Comprehensive Survey of Grammatical Error Correction (2021, methodology): “Besides the mentioned works, many other GEC systems also incorporate monolingual data via back translation [48, 70, 74].”
Neural Grammatical Error Correction Systems with Unsupervised Pre-training on Synthetic Data (2019, methodology): “Ge et al. (2018a) proposed ﬂuency-boost learning that generates additional training examples during training from an independent backward model or the forward model being trained.”
Encode, Tag, Realize: High-Precision Text Editing (2019, methodology): “Neural encoderdecoder methods are also commonly applied to the error correction task (Ge et al., 2018; Chollampatt and Ng, 2018; Zhao et al., 2019), but suffer from a lack of training data, which is why taskspecific tricks need to be applied (Kasewa et al., 2018; Junczys-Dowmunt et al., 2018).”
Reaching Human-level Performance in Automatic Grammatical Error Correction: An Empirical Study (2018, background): “m/en-us/services/cognitive-services/ spell-check/ 9In this report, we do not present a detailed comparison and analysis for different ﬂuency boost learning and inference methods which can be found in Ge et al. (2018). 8 Microsoft Research Technical Report System CoNLL-2014 CoNLL-10 JFLEG test F 0:5 ”
Parallel Iterative Edit Models for Local Sequence Transduction (2019, methodology): “This can either be multiple layers of bidirectional RNNs, CNNs or deep bidirectional transformers.”
CoCoNuT: combining context-aware neural translation models using ensemble for program repair (2020, methodology): “Grammatical Error Correction (GEC): Recent work uses machine translation to fix grammar errors [14, 15, 18, 19, 30, 32, 51, 63, 72, 74, 90].”
Sequence-to-sequence Pre-training with Data Augmentation for Sentence Rewriting (2019, methodology): “To address this challenge, we use ﬂuency discrimination , whose idea was ﬁrst proposed by Ge et al. (2018a).”
Controllable data synthesis method for grammatical error correction (2019, methodology): “Since Brockett(Brockett et al., 2006) introduced the idea of translation to solve the problem of GEC by translating error text into right one, a lot of sequence to sequence models have been applied to GEC (Chollampatt and Ng, 2018a,b; Grundkiewicz and Junczys-Dowmunt, 2018; Ge et al., 2018).”
Improving the Efficiency of Grammatical Error Correction with Erroneous Span Detection and Correction (2020, unknown): “…not only take time to rescore but also slow down the correction model with a larger beam size during inference; multi-round (iterative) decoding (Ge et al., 2018a,b; Lichtarge et al., 2019) needs to repeatedly run the model; BERT-fuse (Kaneko et al., 2020) adds extra computation for model…”
A Neural Grammatical Error Correction System Built On Better Pre-training and Sequential Transfer Learning (2019, methodology): “…al., 2018; Lichtarge et al., 2018; Zhao et al., 2019) made use of the Transformer (Vaswani et al., 2017) architecture and saw results nearly as good as or better than convolutional (Chollampatt and Ng, 2018a,b) and recurrent (Grundkiewicz and Junczys-Dowmunt, 2018; Ge et al., 2018a) architectures.”