Ground Truth for Grammatical Error Correction Metrics

By Courtney Napoles, Keisuke Sakaguchi, Matt Post, Joel R. Tetreault. 2015

238 citations recorded by the source corpus.

How do we know which grammatical error correction (GEC) system is best? A number of metrics have been proposed over the years, each motivated by weaknesses of previous metrics; however, the metrics themselves have not been compared to an empirical gold standard grounded in human judgments. We conducted the ﬁrst human evaluation of GEC system outputs, and show that the rankings produced by metrics such as MaxMatch and I-measure do not correlate well with this ground truth. As a step towards better metrics, we also propose GLEU, a simple variant of BLEU, modiﬁed to account for both the source and the reference, and show that it hews much more closely to human judgments.

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

Citations in the corpus

PEER: A Collaborative Language Model (2022, background): “• SARI (Xu et al., 2016) averages match scores for the three word-level edit operations add, delete and keep;4 • GLEU (Napoles et al., 2015) is a variant of BLEU (Papineni et al., 2002) proposed for grammatical error correction tasks; • Rouge (Lin, 2004) is a set of metrics based on n-gram overlap…”
Optimizing Statistical Machine Translation for Text Simplification (2016, background): “…metrics exist for simpliﬁca-tion, although there are several designed for other text-to-text generation tasks: grammatical error correction (Napoles et al., 2015; Felice and Briscoe, 2015; Dahlmeier and Ng, 2012), paraphrase generation (Chen and Dolan, 2011; Xu et al., 2012; Sun and Zhou,…”
A Multilayer Convolutional Encoder-Decoder Neural Network for Grammatical Error Correction (2018, methodology): “A smaller subset of CC (2.2B words) was used in (Xie et al. 2016) compared to the rest (94B – 97B words). sess ﬂuency of corrected text when the error-span and error-type annotations are not provided (Napoles et al. 2015).”
“Transforming” Delete, Retrieve, Generate Approach for Controlled Text Style Transfer (2019, methodology): “In the interest of space, we omit writing the elaborate equations and explanation for GLEU in this paper, but instead point the reader to Section 4 of Napoles et al. (2015) for the same.”
On Variational Learning of Controllable Representations for Text without Supervision (2019, methodology): “Here, we use the implementation of GLEU4 provided by Napoles et al. (2015) to calculate the GLEU score.”
Artificial error generation for translation-based grammatical error correction (2016, background): “The human ranking reported by the authors differs from the one in Napoles et al. (2015) (see Table 3.”
Prompt-and-Rerank: A Method for Zero-Shot and Few-Shot Arbitrary Textual Style Transfer with Small Language Models (2022, methodology): “…(r-sBLEU) and self - SacreBLEU (s-sBLEU) scores.23 When evaluating the performances of models on the JFLEG corpus, we also used the sentence-level GLEU metric (Napoles et al., 2015), a variant of BLEU that was specifically designed for evaluating grammatical error correction (GEC) models.”
Reaching Human-level Performance in Automatic Grammatical Error Correction: An Empirical Study (2018, methodology): “5 (Dahlmeier & Ng, 2012a) for CoNLL-2014 and use GLEU (Napoles et al., 2015) for JFLEG evaluation.”
A Comprehensive Survey of Grammar Error Correction (2020, background): “The reward 𝑟(𝑦̂,𝑦) is specified as GLEU.”
Improving the Efficiency of Grammatical Error Correction with Erroneous Span Detection and Correction (2020, unknown): “…to the results in these tables, it is clear that our approach can yield satisfying corrections without hurting ﬂuency, which is consistent with our evaluation results in the JFLEG (Napoles et al., 2017) test set with respect to GLEU 7 (Napoles et al., 2015) – an automatic ﬂuency metric for GEC.”
Fluency Boost Learning and Inference for Neural Grammatical Error Correction (2018, methodology): “8 The recently proposed SMT-NMT hybrid system (Grundkiewicz and Junczys-Dowmunt, 2018), which is tuned towards GLEU on JFLEG Dev set, reports a higher result (GLEU=61.50 on JFLEG test set). or MT-based (Brockett et al., 2006; Dahlmeier and Ng, 2011, 2012a; Yoshimoto et al., 2013; Yuan and Felice, 20”
A Comprehensive Survey of Grammatical Error Correction (2021, background): “Since traditional precision, recall, and F-score can be misleading [17], some evaluation metrics have been proposed, including M(2) [33], I [39], GLEU [97], and ERRANT [13].”
Encoder-Decoder Models Can Benefit from Pre-trained Masked Language Models in Grammatical Error Correction (2020, methodology): “We used ERRANT evaluation metrics (Felice et al., 2016; Bryant et al., 2017) for W&I-test, M 2 score (Dahlmeier and Ng, 2012) for CoNLL-2014 and FCE-test sets, and GLEU (Napoles et al., 2015) for JFLEG.”
CGMH: Constrained Sentence Generation by Metropolis-Hastings Sampling (2018, methodology): “We adopted the GLEU metric (Napoles et al. 2015), which measures sentence fluency and grammaticality.”
JFLEG: A Fluency Corpus and Benchmark for Grammatical Error Correction (2017, methodology): “We evaluate system output against the four sets of JFLEG corrections with GLEU, an automatic fluency metric specifically designed for this task (Napoles et al., 2015) and the Max-Match metric (M2) (Dahlmeier and Ng, 2012).”
The BEA-2019 Shared Task on Grammatical Error Correction (2019, methodology): “Note that the I-measure and GLEU are unaffected by the differences between gold and auto references and so are only reported once in this table.”
Deep Just-In-Time Inconsistency Detection Between Comments and Source Code (2020, methodology): “In fact, this can even beat Panthaplackel et al. (2020b) on xMatch, ME-TEOR, BLEU-4, and SARI, and GLEU.”
Sequence-to-sequence Pre-training with Data Augmentation for Sentence Rewriting (2019, unknown): “In JFLEG, it achieves 63.54 GLEU score which is the new state-of-the-art result, even out-performing the multi-round decoding results of Grundkiewicz and Junczys-Dowmunt (2018) and Lichtarge et al. (2019).”
Automatic Annotation and Evaluation of Error Types for Grammatical Error Correction (2017, background): “Since no scorer is currently capable of calculating error type performance however (Dahlmeier and Ng, 2012; Felice and Briscoe, 2015; Napoles et al., 2015), we instead built our own.”
Neural Grammatical Error Correction Systems with Unsupervised Pre-training on Synthetic Data (2019, result): “We also report results on the JFLEG test set (Napoles et al., 2017) using GLEU (Napoles et al., 2015).”