Better Evaluation for Grammatical Error Correction

By Daniel Dahlmeier, Hwee Tou Ng. 2012

408 citations recorded by the source corpus.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding (2021, background): “, 2014) dataset that contains 1,312 sentences as our main test set, and evaluate the speedup as well as MaxMatch (Dahlmeier and Ng, 2012) precision, recall and F0.”
Optimizing Statistical Machine Translation for Text Simplification (2016, background): “…there are several designed for other text-to-text generation tasks: grammatical error correction (Napoles et al., 2015; Felice and Briscoe, 2015; Dahlmeier and Ng, 2012), paraphrase generation (Chen and Dolan, 2011; Xu et al., 2012; Sun and Zhou, 2012), and conversation generation (Gal-ley et…”
Erroneous data generation for Grammatical Error Correction (2019, methodology): “To compare with previous state-of-the-art systems, we provide results of MaxMatch (M 2 ) Scorer (Dahlmeier and Ng, 2012) on the CoNLL-2014 test set.”
Adapting Sequence Models for Sentence Correction (2017, positive): “, 2016) and MaxMatch (M2) metrics (Dahlmeier and Ng, 2012).”
A Comprehensive Survey of Grammar Error Correction (2020, neutral): “Since traditional precision, recall and F-score can be misleading [127], [15], some evaluation metrics have been proposed, including 𝑀 2 [128], 𝐼 [129], GLEU [126] and ERRANT [11].”
Artificial error generation for translation-based grammatical error correction (2016, methodology): “49 3.1.3 Evaluation in CoNLL 2013 & 2014: M2 Scorer . . . . . . . . 50 3.2 Towards a new evaluation method: I-measure . . . . . . . . . . . . . 52 3.2.1 Annotation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 3.2.2 Alignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 ”
Corpora Generation for Grammatical Error Correction (2019, methodology): “5 with the M 2 scorer (Dahlmeier and Ng, 2012b) on the CoNLL-2014 dev and test sets, and the GLEU+ metric (Napoles et al.”
Triple-to-Text: Converting RDF Triples into High-Quality Natural Languages via Optimizing an Inverse KL Divergence (2019, background): “lap (including BLEU, TER [47], ROUGE [36], NIST [17], LEPOR [25], CIDER [52] and METEOR [5]) to semantic similarity like Semantic Text Similarity [26]. Grammar-based metrics include F-score, MaxMatch [12], I-measure [21]. Besides, instead of comparing sentences words by words, EmbSim [57] compares t”
Neural Grammatical Error Correction Systems with Unsupervised Pre-training on Synthetic Data (2019, methodology): “, 2013) calculated with the official M2Scorer (Dahlmeier and Ng, 2012).”
Encoder-Decoder Models Can Benefit from Pre-trained Masked Language Models in Grammatical Error Correction (2020, methodology): “, 2017) for W&I-test, M2 score (Dahlmeier and Ng, 2012) for CoNLL-2014 and FCE-test sets, and GLEU (Napoles et al.”
The BEA-2019 Shared Task on Grammatical Error Correction (2019, methodology): “We also note that despite previous research that suggested MaxMatch correlates more strongly with human judgements than the I-measure (cf. Section 5), the I-measure still ranked the top 10 Restricted Track systems in exactly the same order as MaxMatch F 0 .”
Ground Truth for Grammatical Error Correction Metrics (2015, background): “Thus Dahlmeier and Ng (2012) proposed the MaxMatch (M 2 ) scorer, which calculates the F-score over an edit lattice that captures phrase-level edits.”
Controllable data synthesis method for grammatical error correction (2019, methodology): “We choose the test set from the CoNLL 2013 shared task (Ng et al., 2013) as our development set and report MaxMatch(M2) score (Dahlmeier and Ng, 2012) on the CoNLL-2014 shared task (Ng et al., 2014) test set.”
Fluency Boost Learning and Inference for Neural Grammatical Error Correction (2018, background): “…error detection (Leacock et al., 2010; Rei and Yannakoudakis, 2016; Kaneko et al., 2017) and GEC evaluation (Tetreault et al., 2010b; Madnani et al., 2011; Dahlmeier and Ng, 2012c; Napoles et al., 2015; Sakaguchi et al., 2016; Napoles et al., 2016; Bryant et al., 2017; Asano et al., 2017).”
A Comprehensive Survey of Grammatical Error Correction (2021, neutral): “Since traditional precision, recall, and F-score can be misleading [17], some evaluation metrics have been proposed, including M(2) [33], I [39], GLEU [97], and ERRANT [13].”
Using Wikipedia Edits in Low Resource Grammatical Error Correction (2018, methodology): “8 We train a single model for each condition and evaluate on the Falko-MERLIN test set using M 2 scorer (Dahlmeier and Ng, 2012).”
Automatic Annotation and Evaluation of Error Types for Grammatical Error Correction (2017, methodology): “, 2011) is unlikely to perform as well on the National University of Singapore Corpus of Learner English (NUCLE) (Dahlmeier and Ng, 2012) or vice versa, because both corpora have been annotated according to different standards (cf.”
Gender-Aware Reinflection using Linguistically Enhanced Neural Models (2020, methodology): “Additionally, we use the MaxMatch (M 2 ) scorer (Dahlmeier and Ng, 2012) to compute the word-level edits between the input and reinﬂected output.”
Improving Grammatical Error Correction with Data Augmentation by Editing Latent Representation (2020, methodology): “We report results on CoNLL-2014 benchmark evaluated by official M2 scorer(Dahlmeier and Ng, 2012), and on BEA-2019 and FCE benchmarks evaluated by ERRANT.”
Learning to Model Editing Processes (2022, background): “For instance, there are other widely acknowledged editing tasks in NLP such as grammatical error correction (Dahlmeier and Ng, 2012) or style transfer (Shen et al., 2017).”