A Multilayer Convolutional Encoder-Decoder Neural Network for Grammatical Error Correction

By Shamil Chollampatt, Hwee Tou Ng. 2018

234 citations recorded by the source corpus.

We improve automatic correction of grammatical, orthographic, and collocation errors in text using a multilayer convolutional encoder-decoder neural network. The network is initialized with embeddings that make use of character N-gram information to better suit this task. When evaluated on common benchmark test data sets (CoNLL-2014 and JFLEG), our model substantially outperforms all prior neural approaches on this task as well as strong statistical machine translation-based systems with neural and task-specific features trained on the same data. Our analysis shows the superiority of convolutional neural networks over recurrent neural networks such as long short-term memory (LSTM) networks in capturing the local context via attention, and thereby improving the coverage in correcting grammatical errors. By ensembling multiple models, and incorporating an N-gram language model and edit features via rescoring, our novel method becomes the first neural approach to outperform the current state-of-the-art statistical machine translation-based approach, both in terms of grammaticality and fluency.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

fairseq: A Fast, Extensible Toolkit for Sequence Modeling (2019, background): “, 2018b, 2019), error correction (Chollampatt and Ng, 2018), multilingual sentence embeddings (Artetxe and Schwenk, 2018), and dialogue (Miller et al.”
Corpora Generation for Grammatical Error Correction (2019, methodology): “tion (NMT) model to reach the state of the art on CoNLL-2014 (Chollampatt and Ng, 2018) used an ensemble of four convolutional sequence-tosequence models followed by rescoring.”
A Neural Grammatical Error Correction System Built On Better Pre-training and Sequential Transfer Learning (2019, unknown): “, 2014) has been different adaptations of phrase-based and statistical machine translation (PBSMT) models (Junczys-Dowmunt and Grundkiewicz, 2016), more recent work on GEC increasingly adopted partial (Grundkiewicz and JunczysDowmunt, 2018) or exclusive (Junczys-Dowmunt et al., 2018; Chollampatt and”
CoCoNuT: combining context-aware neural translation models using ensemble for program repair (2020, methodology): “Grammatical Error Correction (GEC): Recent work uses machine translation to fix grammar errors [14, 15, 18, 19, 30, 32, 51, 63, 72, 74, 90].”
Controllable data synthesis method for grammatical error correction (2019, methodology): “encoder-decoder neural network which outperforms the current state-of-the-art statistical machine translation-based approach (Chollampatt and Ng, 2018a).”
Reaching Human-level Performance in Automatic Grammatical Error Correction: An Empirical Study (2018, methodology): “For singleround inference, we follow Chollampatt & Ng (2018) to generate 12-best predictions and choose the best sentence after re-ranking with edit operation and language model scores.”
Using Wikipedia Edits in Low Resource Grammatical Error Correction (2018, methodology): “We create a new GEC corpus for German along with the models needed for the neural GEC approach presented in Chollampatt and Ng (2018).”
The Unreasonable Effectiveness of Transformer Language Models in Grammatical Error Correction (2019, methodology): “, 2017) and neural (Ji et al., 2017; Xie et al., 2016; Yuan and Briscoe, 2016; Chollampatt et al., 2016; Chollampatt and Ng, 2017; Sakaguchi et al., 2017; Chollampatt and Ng, 2018) machine translation methods for GEC, we follow the approach of Bryant and Briscoe (2018) and explore how such models wo”
A Comprehensive Survey of Grammatical Error Correction (2021, methodology): “Many NMT-based GEC systems utilize a spellchecker(1) or open-source spell checking library(2) in preprocessing [22, 26, 48, 50, 67, 116].”
Noising and Denoising Natural Language: Diverse Backtranslation for Grammar Correction (2018, methodology): “Token noising performs surprisingly well; we suspect this is because a significant portion of errors in the JFLEG dataset are spelling errors, as demonstrated from strong gains in performance by using a spelling checker reported by Chollampatt and Ng (2018).”
Erroneous data generation for Grammatical Error Correction (2019, methodology): “Chollampatt and Ng (2018) used a convolutional sequence-to-sequence (seq2seq) model (Gehring et al., 2017) with a large language model for rescoring.”
Prediction of aerodynamic flow fields using convolutional neural networks (2019, background): “Encoder–decoder CNNs are most widely used for machine translation from a source language to a target language [6].”
Tree-Transformer: A Transformer-Based Method for Correction of Tree-Structured Data (2019, methodology): “Chollampatt and Ng [7] created a convolution-based encoder and decoder network which was the ﬁrst to beat state of the art phrased-based systems.”
Fluency Boost Learning and Inference for Neural Grammatical Error Correction (2018, background): “, 2014) for grammatical error correction (GEC) have drawn growing attention (Yuan and Briscoe, 2016; Xie et al., 2016; Ji et al., 2017; Schmaltz et al., 2017; Sakaguchi et al., 2017; Chollampatt and Ng, 2018) in recent years.”
Sequence-to-sequence Pre-training with Data Augmentation for Sentence Rewriting (2019, methodology): “As most of the systems (Sak-aguchi et al., 2017; Chollampatt and Ng, 2018a; Grundkiewicz and Junczys-Dowmunt, 2018) that use an additional spell checker to resolve spelling errors in JFLEG, we follow (Ge et al., 2018a) to use the public Bing spell checker 2 to ﬁx spelling errors in JFLEG as…”
Neural Grammatical Error Correction Systems with Unsupervised Pre-training on Synthetic Data (2019, methodology): “Work started with statistical phrase-based machine translation (SMT) methods (Junczys-Dowmunt and Grundkiewicz, 2016; Chollampatt and Ng, 2017) while sequence-to-sequence methods adopted from neural machine translation (NMT) lagged in quality until recently (Chollampatt and Ng, 2018; JunczysDowmunt ”
Seq2Edits: Sequence Transduction Using Span-level Edit Operations (2020, methodology): “This resembles the inverse setup of Chollampatt and Ng (2018) who used edit features to rescore a full sequence model.”
Gender-Aware Reinflection using Linguistically Enhanced Neural Models (2020, unknown): “Sequence-to-sequence models have achieved signiﬁcant results in grammatical error correction (GEC) (Chollampatt and Ng, 2018; Junczys-Dowmunt et al., 2018; Grundkiewicz et al., 2019) and morphological reinﬂection tasks (Faruqui et al., 2016; Kann and Schütze, 2016; Aharoni and Goldberg, 2017).”
The BEA-2019 Shared Task on Grammatical Error Correction (2019, methodology): “…systems have been trained, tuned and tested on different combinations of metrics and corpora (Sakaguchi et al., 2017; Yannakoudakis et al., 2017; Chollampatt and Ng, 2018a; Ge et al., 2018; Grundkiewicz and Junczys-Dowmunt, 2018; Junczys-Dowmunt et al., 2018; Lichtarge et al., 2018; Zhao et…”