BLEU is Not Suitable for the Evaluation of Text Simplification

By Elior Sulem, Omri Abend, Ari Rappoport. 2018

225 citations recorded by the source corpus.

BLEU is widely considered to be an informative metric for text-to-text generation, including Text Simplification (TS). TS includes both lexical and structural aspects. In this paper we show that BLEU is not suitable for the evaluation of sentence splitting, the major structural simplification operation. We manually compiled a sentence splitting gold standard corpus containing multiple structural paraphrases, and performed a correlation analysis with human judgments. We find low or no correlation between BLEU and the grammaticality and meaning preservation parameters where sentence splitting is involved. Moreover, BLEU often negatively correlates with simplicity, essentially penalizing simpler sentences.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

ColdGANs: Taming Language GANs with Cautious Sampling Strategies (2020, background): “However, such metrics, based on n -grams similarity, are known to poorly correlate with human judgments [28], and do not preserve meaning [40].”
Discriminative Adversarial Search for Abstractive Summarization (2020, neutral): “Still, these automated metrics suffer from known limitations: Sulem et al. (2018) showed how BLEU metrics do not reﬂect meaning preservation, while Novikova et al. (2017) pointed out that, for NLG tasks, they do not map well to human judgements.”
Language as an Abstraction for Hierarchical Deep Reinforcement Learning (2019, background): “HER [3]), which uses a state vector or a taskrelevant part of the state vector as the goal, it is difficult to define meaningful distance metrics in the space of language statements [7, 47, 55], and, consequently, difficult to make the reward signal smooth by assigning partial credit (unlike, e.”
GEMv2: Multilingual NLG Benchmarking in a Single Line of Code (2022, methodology): “As the ﬁrst challenge set, we include the HSPLIT-Wiki test set, containing 359 pairs (Sulem et al., 2018).”
A Comprehensive Assessment of Dialog Evaluation Metrics (2021, background): “While BLEU performs reasonably when evaluating NLG systems, it has issues reﬂecting grammaticality, use of semantically similar words, and meaning preservation (Novikova et al., 2017; Ananthakrishnan et al., 2007; Sulem et al., 2018; Reiter, 2018).”
EASSE: Easier Automatic Sentence Simplification Evaluation (2019, methodology): “SAMSA measures structural simplicity (i.e. sentence splitting).”
Synthetic Data Augmentation for Zero-Shot Cross-Lingual Question Answering (2020, background): “This makes automatic evaluation challenging for this setup, as metrics suffer from similar difﬁculties as those observed for text generation (Sulem et al., 2018).”
NUBIA: NeUral Based Interchangeability Assessor for Text Generation (2020, background): “These metrics have posed serious limitations and have already been criticized by the academic community (Reiter, 2018) (Callison-Burch et al., 2006) (Sulem et al., 2018) (Novikova et al., 2017).”
ASSET: A Dataset for Tuning and Evaluation of Sentence Simplification Models with Multiple Rewriting Transformations (2020, methodology): “HSplit (Sulem et al., 2018a), on the other hand, provides simpliﬁcations involving only splitting for sentences in the test set of Turk-Corpus.”