Predicting Grammaticality on an Ordinal Scale

By Michael Heilman, Aoife Cahill, Nitin Madnani, Melissa Lopez, Matthew David Mulholland, Joel R. Tetreault. 2014

91 citations recorded by the source corpus.

Automated methods for identifying whether sentences are grammatical have various potential applications (e.g., machine translation, automated essay scoring, computer-assisted language learning). In this work, we construct a statistical model of grammaticality using various linguistic features (e.g., misspelling counts, parser outputs, n-gram language model scores). We also present a new publicly available dataset of learner sentences judged for grammaticality on an ordinal scale. In evaluations, we compare our system to the one from Post (2011) and find that our approach yields state-of-the-art performance.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

BLiMP: The Benchmark of Linguistic Minimal Pairs for English (2019, neutral): “Corpora of sentences and their gram-maticality are collected for this purpose in a number of studies (Heilman et al., 2014; Lau et al., 2017; Warstadt et al., 2019b).”
Artificial error generation for translation-based grammatical error correction (2016, background): “Firstly, they seem naturally suited for error detection, since we can judge the grammaticality of a sentence or phrase from their estimated probabilities (Heilman et al., 2014; Hernandez and Calvo, 2014; Lee et al., 2014; Lin and Chen, 2015; Okanohara and Tsujii, 2007; Wagner et al., 2007).”
A Comprehensive Survey of Grammar Error Correction (2020, methodology): “The first trial is the research about LFM score [131] and error count score given by e-rater (ER) and Language Tool (LT) [132].”
Dear Sir or Madam, May I Introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfer (2018, methodology): “Meaning Preservation: Following the annotation scheme developed for the Semantic Textual Similarity (STS) dataset (Agirre et al., 2016), given two sentences i.e. the source style sentence and the target style reference rewrite or the target style model output, workers rate the meaning similarity of ”
JFLEG: A Fluency Corpus and Benchmark for Grammatical Error Correction (2017, methodology): “Following the recommendations of Sakaguchi et al. (2016), we release a new corpus for GEC, the JHU FLuency-Extended GUG corpus (JFLEG), which adds a layer of annotation to the GUG corpus (Heilman et al., 2014).”
BLiMP: A Benchmark of Linguistic Minimal Pairs for English (2020, background): “tences and their grammaticality are collected for this purpose in a number of computational studies on grammaticality judgment (Heilman et al., 2014; Lau et al., 2017; Warstadt et al., 2019b).”
Corpora Generation for Grammatical Error Correction (2019, methodology): “, 2014) and the JFLEG test set (Napoles et al., 2017; Heilman et al., 2014).”
Towards a Decomposable Metric for Explainable Evaluation of Text Generation from AMR (2020, background): “Assessing sentence grammaticality and fluency is not an easy task (Heilman et al., 2014; Katinskaia and Ivanova, 2019).”
Formalizing and Benchmarking Prompt Injection Attacks and Defenses (2023, background): “If the target (or injected) task is the grammar correction task, M (or M e ) is the GLEU score [24].”
A Review of Human Evaluation for Style Transfer (2021, background): “Some papers follow Agirre et al. (2016) for measuring textual similarity, Heilman et al. (2014) for grammaticality and Pavlick and Tetreault (2016) for formality.”
Evaluating the Evaluation Metrics for Style Transfer: A Case Study in Multilingual Formality Transfer (2021, background): “A few other works train models on EN data annotated for grammaticality (Heilman et al., 2014) or linguistic acceptability (Warstadt et al., 2019) instead.”