JFLEG: A Fluency Corpus and Benchmark for Grammatical Error Correction

By Joel R. Tetreault, Keisuke Sakaguchi, Courtney Napoles. 2017

231 citations recorded by the source corpus.

We present a new parallel corpus, JHU FLuency-Extended GUG corpus (JFLEG) for developing and evaluating grammatical error correction (GEC). Unlike other corpora, it represents a broad range of language proficiency levels and uses holistic fluency edits to not only correct grammatical errors but also make the original text more native sounding. We describe the types of corrections made and benchmark four leading GEC systems on this corpus, identifying specific areas in which they do well and how they can improve. JFLEG fulfills the need for a new gold standard to properly assess the current state of GEC.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

PEER: A Collaborative Language Model (2022, methodology): “For this evaluation, we consider the following datasets: • JFLEG (Napoles et al., 2017) is a grammatical error correction dataset with singlesentence inputs written by English language learners; • ASSET (Alva-Manchego et al., 2020) is a crowdsourced corpus for single-sentence text simplification; • ”
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks (2022, methodology): “Table 17: An example task in the Grammar Error Correction category of our dataset, adopted from JFLEG (Napoles et al., 2017).”
Noising and Denoising Natural Language: Diverse Backtranslation for Grammar Correction (2018, methodology): “Our test sets are the CoNLL 2014 evaluation set and the JFLEG test set (Ng et al., 2014; Napoles et al., 2017).”
Formalizing and Benchmarking Prompt Injection Attacks and Defenses (2023, methodology): “To address the challenge, we propose to set the threshold as follows: we collect some clean data records; use an LLM to calculate the model-dependent perplexity for each clean data record; and we select a threshold such that the False Positive Rate , i.e., fraction of the clean data records falsely…”
Reaching Human-level Performance in Automatic Grammatical Error Correction: An Empirical Study (2018, methodology): “, 2014) and JFLEG (Napoles et al., 2017) test set as our evaluation datasets.”
Prompt-and-Rerank: A Method for Zero-Shot and Few-Shot Arbitrary Textual Style Transfer with Small Language Models (2022, methodology): “747 (Napoles et al., 2017) Grammatical For example, my cousin is 12 years old.”
Erroneous data generation for Grammatical Error Correction (2019, methodology): “5 with ERRANT (Bryant et al., 2017) on the ABCN test, GLEU (Sakaguchi et al., 2016) on the JFLEG test set (Napoles et al., 2017).”
CGMH: Constrained Sentence Generation by Metropolis-Hastings Sampling (2018, methodology): “We evaluated our method on JFLEG (Napoles, Sakaguchi, and Tetreault 2017), 4 a newly released dataset for sentence correction.”
Parallel Iterative Edit Models for Local Sequence Transduction (2019, methodology): “, 2016) scores on JFLEG corpus (Napoles et al., 2017) to evaluate fluency.”
Sequence-to-sequence Pre-training with Data Augmentation for Sentence Rewriting (2019, methodology): “5 (Dahlmeier and Ng, 2012) for CoNLL-2014 and GLEU (Napoles et al., 2015) for JFLEG evaluation.”
Corpora Generation for Grammatical Error Correction (2019, methodology): “However, the model trained on all data performs best on the JFLEG set, which has a different distribution of errors relative to CoNLL-2014 (Napoles et al., 2017).”
Neural Grammatical Error Correction Systems with Unsupervised Pre-training on Synthetic Data (2019, result): “We also report results on the JFLEG test set (Napoles et al., 2017) using GLEU (Napoles et al.”
Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models (2023, unknown): “Generation benchmarks consider hallucination as a generation characteristic (Lin et al., 2021; Lee et al., 2022; Min et al., 2023; Yu et al., 2023a) , similar to fluency (Napoles et al., 2017) and coherence (Du et al., 2022), and evaluate the generated texts from LLMs.”
A Comprehensive Survey of Grammatical Error Correction (2021, methodology): “Besides, no system is evaluated on JFLEG before it was released.”
Fluency Boost Learning and Inference for Neural Grammatical Error Correction (2018, methodology): “Moreover, we evaluate our approach on JFLEG corpus (Napoles et al., 2017).”
Seq2Edits: Sequence Transduction Using Span-level Edit Operations (2020, methodology): “• Grammatical error correction (Ng et al., 2014; Napoles et al., 2017; Bryant et al., 2019) – correcting grammatical errors in written text, e.g. “In a such situaction”→ “In such a situation”.”
Encoder-Decoder Models Can Benefit from Pre-trained Masked Language Models in Grammatical Error Correction (2020, methodology): “Therefore, we used GEC evaluation data such as W&I-test, CoNLL-2014 (Ng et al., 2014), FCE-test and JF-LEG (Napoles et al., 2017).”
Improving the Efficiency of Grammatical Error Correction with Erroneous Span Detection and Correction (2020, unknown): “According to the results in these tables, it is clear that our approach can yield satisfying corrections without hurting ﬂuency, which is consistent with our evaluation results in the JFLEG (Napoles et al., 2017) test set with respect to GLEU 7 (Napoles et al., 2015) – an automatic ﬂuency metric for”