Automatic Annotation and Evaluation of Error Types for Grammatical Error Correction

By Christopher Bryant, Mariano Felice, Ted Briscoe. Annual Meeting of the Association for Computational Linguistics, 2017

372 citations recorded by the source corpus.

Until now, error type performance for Grammatical Error Correction (GEC) systems could only be measured in terms of recall because system output is not annotated. To overcome this problem, we introduce ERRANT, a grammatical ERRor ANnotation Toolkit designed to automatically extract edits from parallel original and corrected sentences and classify them according to a new, dataset-agnostic, rule-based framework. This not only facilitates error type evaluation at different levels of granularity, but can also be used to reduce annotator workload and standardise existing GEC datasets. Human experts rated the automatic edits as “Good” or “Acceptable” in at least 95% of cases, so we applied ERRANT to the system output of the CoNLL-2014 shared task to carry out a detailed error type analysis for the first time.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/4cac1e1eb876ffdcfda5a62d5237f942b519a502

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Erroneous data generation for Grammatical Error Correction (2019, methodology): “5 with ERRANT (Bryant et al., 2017) on the ABCN test, GLEU (Sakaguchi et al., 2016) on the JFLEG test set (Napoles et al., 2017).”
Improving Grammatical Error Correction with Data Augmentation by Editing Latent Representation (2020, methodology): “LM features include the score of a language model which is trained on the web-scale Common Crawl corpus (Chollampatt and Ng, 2017; Junczys-Dowmunt and Grundkiewicz, 2016), and the length of the output sequences.”
Using Wikipedia Edits in Low Resource Grammatical Error Correction (2018, methodology): “ERRANT, the ERRor ANnotation Tool (Felice et al., 2016; Bryant et al., 2017), analyzes pairs of English sentences from a GEC corpus to identify the types of corrections performed.”
A Neural Grammatical Error Correction System Built On Better Pre-training and Sequential Transfer Learning (2019, methodology): “In the token-based approach, we make use of extracted human edits from annotated GEC corpora, using automated error annotation toolkits such as ERRANT (Bryant et al., 2017).”
A Comparative Study of Synthetic Data Generation Methods for Grammatical Error Correction (2020, methodology): “FCE was originally annotated according to a different error type framework, but was re-annotated automatically using ERRANT for use in the shared task.”
Gender-Aware Reinflection using Linguistically Enhanced Neural Models (2020, methodology): “We are aware that there are other tools to consider for word-level edit calculation such as ERRANT (Bryant et al., 2017), but we did not use them as they require additional dependencies to work for Arabic.”
Reaching Human-level Performance in Automatic Grammatical Error Correction: An Empirical Study (2018, background): “…(Leacock et al., 2010; Rei & Yannakoudakis, 2016; Kaneko et al., 2017) and GEC evaluation (Tetreault et al., 2010b; Madnani et al., 2011; Dahlmeier & Ng, 2012c; Napoles et al., 2015; Sakaguchi et al., 2016; Napoles et al., 2016; Bryant et al., 2017; Asano et al., 2017; Choshen & Abend, 2018).”
A Comprehensive Survey of Grammatical Error Correction (2021, methodology): “ERRANT is an improved version ofM2 scorer.”
Improving Grammatical Error Correction with Machine Translation Pairs (2019, methodology): “We then conduct an in-depth analysis of the errortype contrained in the synthetic data generated by our approach and other data synthesis methods using ERRANT (Bryant et al., 2017).”
GECToR – Grammatical Error Correction: Tag, Not Rewrite (2020, methodology): “We report results on CoNLL-2014 test set (Ng et al., 2014) evaluated by ofﬁ-cial M 2 scorer (Dahlmeier and Ng, 2012), and on BEA-2019 dev and test sets evaluated by ERRANT (Bryant et al., 2017).”
Controllable data synthesis method for grammatical error correction (2019, methodology): “We use the ERRANT6 to evaluate the model performance by different error types.”
Fluency Boost Learning and Inference for Neural Grammatical Error Correction (2018, neutral): “…error detection (Leacock et al., 2010; Rei and Yannakoudakis, 2016; Kaneko et al., 2017) and GEC evaluation (Tetreault et al., 2010b; Madnani et al., 2011; Dahlmeier and Ng, 2012c; Napoles et al., 2015; Sakaguchi et al., 2016; Napoles et al., 2016; Bryant et al., 2017; Asano et al., 2017).”
Encoder-Decoder Models Can Benefit from Pre-trained Masked Language Models in Grammatical Error Correction (2020, methodology): “We use ERRANT (Felice et al., 2016; Bryant et al., 2017) to measure F 0 .”
The BEA-2019 Shared Task on Grammatical Error Correction (2019, methodology): “is calculated using the ERRANT scorer (Bryant et al., 2017), rather than the M2 scorer (Dahlmeier and Ng, 2012), because the ERRANT scorer can provide much more detailed feedback, e.”
Neural Grammatical Error Correction Systems with Unsupervised Pre-training on Synthetic Data (2019, methodology): “The performance of participating systems was evaluated using the ERRANT scorer (Bryant et al., 2017) which reports a F0.”
Seq2Edits: Sequence Transduction Using Span-level Edit Operations (2020, methodology): “ERRANT (Bryant et al., 2017; Felice et al., 2016) to derive span-level edits from the parallel text.”
A Comprehensive Survey of Grammar Error Correction (2020, background): “It is worthy to mention that system with the highest ERRANT score (73.18) [105] and system with the fourth highest score (66.78) [84] on BEA2019 test set are based on sophisticated ensemble strategies combining the output of different GEC models, highlighting the benefit brought by combining differe”
A Multilayer Convolutional Encoder-Decoder Neural Network for Grammatical Error Correction (2018, methodology): “Although neural network (NN) models have been used as features to improve the generalization of the SMT approach (Chollampatt, Taghipour, and Ng 2016), SMT still suffers from limitations in accessing the global source and target context effectively.”