Revisiting Readability: A Unified Framework for Predicting Text Quality

By Emily Pitler, Ani Nenkova. 2008

510 citations recorded by the source corpus.

We combine lexical, syntactic, and discourse features to produce a highly predictive model of human readers' judgments of text readability. This is the first study to take into account such a variety of linguistic factors and the first to empirically demonstrate that discourse relations are strongly associated with the perceived quality of text. We show that various surface metrics generally expected to be related to readability are not very good predictors of readability judgments in our Wall Street Journal corpus. We also establish that readability predictors behave differently depending on the task: predicting text readability or ranking the readability. Our experiments indicate that discourse relations are the one class of features that exhibits robustness across these two tasks.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Measuring Data (2022, background): “Readability (Flesch, 1979; Halliday, 1989) (language ): Readability aims to represent how difﬁcult a text is to a reader, and is frequently used in domains such as language didactics (To et al., 2013) and corpus linguistics (Pitler and Nenkova, 2008).”
Position paper: Moving task-based language teaching forward (2017, background): “However, in more recent work (e.g. Pitler & Nenkova, 2008) measures that also take account of discourse factors such as cohesion and coherence have been developed.”
General Purpose Text Embeddings from Pre-trained Language Models for Scalable Inference (2020, background): “…predict its topic (Zhang et al., 2015), sentiment (Pang and Lee, 2004; Maas et al., 2011; Socher et al., 2013; Zhang et al., 2015), overall text quality (Pitler and Nenkova, 2008), whether it is humorous (Yang et al., 2015) or offensive (Schmidt and Wiegand, 2017; Zampieri et al., 2019) and so on.”
Story Comprehension for Predicting What Happens Next (2017, background): “2004; Fung and Ngai, 2006), co-occurrence of words (Lapata, 2003; Soricut and Marcu, 2006), syntactic patterns (Louis and Nenkova, 2012) and discourse relations (Pitler and Nenkova, 2008; Lin et al., 2011).”
Problems in Current Text Simplification Research: New Data Can Help (2015, background): “In addition, readability is widely studied and can be automatically estimated (Kincaid et al., 1975; Pitler and Nenkova, 2008; Petersen and Ostendorf, 2009).”
Best practices for the human evaluation of automatically generated text (2019, methodology): “…criteria can be assessed using more objective measures, similar to existing readability measures (e.g., Ambati et al., 2016; Kincaid et al., 1975; Pitler and Nenkova, 2008; Vajjala and Meurers, 2014), where objective text metrics (e.g. average word length, average parse tree height, average…”
The effect of wording on message propagation: Topic- and author-controlled natural experiments on Twitter (2014, background): “Predicting the “success” of various texts such as novels and movie quotes has been the aim of additional prior work not already mentioned in §1 (Ashok et al., 2013; Louis and Nenkova, 2013; Danescu-Niculescu-Mizil et al., 2012; Pitler and Nenkova, 2008; McIntyre and Lapata, 2009).”
Learning the Curriculum with Bayesian Optimization for Task-Specific Word Representation Learning (2016, methodology): “Our features are inspired by prior research in second language acquisition, text simplification, and readability assessment (Schwarm and Ostendorf, 2005; Heilman et al., 2007; Pitler and Nenkova, 2008; Vajjala and Meurers, 2012).”