Why We Need New Evaluation Metrics for NLG

By Jekaterina Novikova, Ondrej Dusek, Amanda Cercas Curry, Verena Rieser. 2017

529 citations recorded by the source corpus.

The majority of NLG evaluation relies on automatic metrics, such as BLEU . In this paper, we motivate the need for novel, system- and data-independent automatic evaluation methods: We investigate a wide range of metrics, including state-of-the-art word-based and novel grammar-based ones, and demonstrate that they only weakly reflect human judgements of system outputs as generated by data-driven, end-to-end NLG. We also show that metric performance is data- and system-specific. Nevertheless, our results also suggest that automatic metrics perform reliably at system-level and can support system development by finding cases where a system performs poorly.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Automatic Detection of Generated Text is Easiest when Humans are Fooled (2019, background): “automatic evaluation of generative model quality (Novikova et al., 2017; Kannan and Vinyals, 2017; Lowe et al., 2017).”
Towards a Human-like Open-Domain Chatbot (2020, background): “BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), or other related metrics in translation and summarization, while popular and easy to compute, have been shown to be unsuitable for dialog (Liu et al., 2016) or more broadly language generation systems (Novikova et al., 2017).”
Neural Generation for Czech: Data and Baselines (2019, methodology): “To obtain a deeper insight into the results and account for automatic metrics’ inaccuracy (Novikova et al., 2017; Reiter, 2018), we performed a detailed manual error analysis on a sample of 100 outputs produced by all systems except the ones with random baseline lexicalizers, which clearly perform…”
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, background): “…metrics based on word-overlap with reference responses, as frequently used in question-answering (Rajpurkar et al., 2016) or machine translation (Papineni et al., 2002), do not work well, and have poor correlation with human judgments (Liu et al., 2016; Novikova et al., 2017; Lowe et al., 2017).”
ColdGANs: Taming Language GANs with Cautious Sampling Strategies (2020, background): “However, such metrics, based on n -grams similarity, are known to poorly correlate with human judgments [28], and do not preserve meaning [40].”
Discriminative Adversarial Search for Abstractive Summarization (2020, neutral): “Still, these automated metrics suffer from known limitations: Sulem et al. (2018) showed how BLEU metrics do not reﬂect meaning preservation, while Novikova et al. (2017) pointed out that, for NLG tasks, they do not map well to human judgements.”
How Decoding Strategies Affect the Verifiability of Generated Text (2019, background): “Several metrics have been proposed to evaluate natural language generations in the past (Novikova et al., 2017).”
ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons (2019, background): “This means standard metrics for tasks like question-answering or machine translation do not work well, and have poor correlation with human judgments (Liu et al. 2016; Novikova et al. 2017).”
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering (2018, background): “However, the reliability of these metrics is questionable because they have been shown to correlate poorly with human judgement (Novikova et al., 2017).”
Better Rewards Yield Better Summaries: Learning to Summarise Without References (2019, background): “…correlation between multiple widely used summary evaluation metrics and human judgement scores, so as to further motivate the need for a stronger reward for RL. Metrics we consider include ROUGE (full length F-score), BLEU (Papineni et al., 2002) and METEOR (Lavie and Denkowski, 2009).”
Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluators (2019, background): “, 2010), the problem is aggravated for dialog generation evaluation because of the much larger output space (Liu et al., 2016; Novikova et al., 2017).”
DART: Open-Domain Structured Data Record to Text Generation (2020, background): “However, these measures have limitations in considering the semantic meanings of words or phrases, which is shown in Novikova et al. (2017a) that these measures fail to correlate well with human judgement.”
Measuring Data (2022, methodology): “It has been proposed as a metric for evaluating natural language generation (Novikova et al., 2017) but has yet to be used for dataset analysis, which could help determine the complexity and linguistic density of texts used for training models.”
Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators (2024, background): “In closed-ended tasks, such as multiple-choice QA, such evaluations are straightforward to implement and trust (Novikova et al., 2017; Yeh et al., 2021).”
BLEURT: Learning Robust Metrics for Text Generation (2020, background): “Thus, they have been repeatedly shown to correlate poorly with human judgment, in particular when all the systems to compare have a similar level of accuracy (Liu et al., 2016; Novikova et al., 2017; Chaganty et al., 2018).”
What makes a good conversation? How controllable attributes affect human judgments (2019, background): “While word-overlap metrics are effective for questionanswering and machine translation, for dialogue they have little to no correlation with human judgments (Liu et al., 2016; Novikova et al., 2017) – this is due to the open-ended nature of dialogue.”
Unifying Human and Statistical Evaluation for Natural Language Generation (2019, background): “Automatic metrics such as BLEU (Papineni et al., 2002) and ROUGE (Lin and Rey, 2004) capture quality better than perplexity but still correlate poorly with human evaluation and fail to capture diversity (Novikova et al., 2017; Chaganty et al., 2018).”