A Survey of Evaluation Metrics Used for NLG Systems
By Ananya B. Sai, Akash Kumar Mohankumar, Mitesh M. Khapra. 2020
344 citations recorded by the source corpus.
In the last few years, a large number of automatic evaluation metrics have been proposed for evaluating Natural Language Generation (NLG) systems. The rapid development and adoption of such automatic evaluation metrics in a relatively short time has created the need for a survey of these metrics. In this survey, we (i) highlight the challenges in automatically evaluating NLG systems, (ii) propose a coherent taxonomy for organising existing evaluation metrics, (iii) briefly describe different existing metrics, and finally (iv) discuss studies criticising the use of automatic evaluation metrics. We then conclude the article highlighting promising future directions of research.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Natural Language Generation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yejin Choi
- Percy Liang
- Mohit Bansal
- Luke S. Zettlemoyer
- Jianfeng Gao
- Noah A. Smith
- Graham Neubig
- Hannaneh Hajishirzi
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Bernt Schiele
- Xiaogang Wang
- Jitendra Malik
- Ming-Hsuan Yang
- Alan L. Yuille
In Natural Language Processing
- Yejin Choi
- Noah A. Smith
- Mohit Bansal
- Swabha Swayamdipta
- Luke S. Zettlemoyer
- Sean Welleck
- Rowan Zellers
- Jianfeng Gao
In Natural Language Generation
Citations in the corpus
- Measuring Massive Multitask Language Understanding (2020, background): “However, NLG is notoriously difficult to evaluate and lacks a standard metric (Sai et al., 2020).”
- From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge (2024, neutral): “However, metric-based evaluation overemphasizes lexical overlap and similarity, which may fall short when many valid responses and more nuanced semantic attributes are required to be considered (Post, 2018; Sai et al., 2022).”
- Active Evaluation: Efficient NLG Evaluation with Few Pairwise Comparisons (2022, background): “In particular, 042 many studies have shown that they do not correlate 043 well with human judgments (Novikova et al., 2017; 044 Elliott and Keller, 2014; Sai et al., 2019, 2020a,b).”
- Play the Shannon Game With Language Models: A Human-Free Approach to Summary Evaluation (2021, neutral): “Beyond summarization, there have been many metrics proposed for Natural Language Generation more generally (Sai, Mohankumar, and Khapra 2020).”
- Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation (2023, background): “Similarly, Sai et al. (2022) discuss how paraphrase detection can be used to evaluate NLG and other related methods might transfer to uncertainty estimation.”
- Human Evaluation and Correlation with Automatic Metrics in Consultation Note Generation (2022, positive): “Following the taxonomies reported by Celikyilmaz et al. (2020) and Sai et al. (2020), the metrics considered can be loosely grouped in: • Text overlap metrics.”
- Towards a Decomposable Metric for Explainable Evaluation of Text Generation from AMR (2020, background): “While current NLG metrics lack interpretability and mainly focus on the form of generated text (Sai et al., 2020), in this work we emphasize the meaning aspect in NLG evaluation, which is most clearly dissociated from form when generating text from structured inputs such as AMR.”
- AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap (2023, background): “We note that the ML and natural language processing (NLP) communities have long dealt with the challenges of evaluating the performance of generative models [164].”
- Unleashing the potential of prompt engineering in Large Language Models: a comprehensive review (2023, background): “[101] Subjective evaluation and objective evaluation methods each have their own advantages and disadvantages.”
- Perturbation CheckLists for Evaluating NLG Evaluation Metrics (2021, methodology): “In case of dialogue, several n-gram-based and embedding-based metrics have been shown to fall short in capturing the diversity of the valid responses (Liu et al., 2016; Sai et al., 2020a).”
- Evaluation of Retrieval-Augmented Generation: A Survey (2024, neutral): “Searches in a dynamic database may lead to misleading results or contradicting the facts.”
- Automatic Text Evaluation through the Lens of Wasserstein Barycenters (2021, background): “Designing better summarization metrics is an active area of research (Scialom et al., 2021) and many of these metrics can be further optimized to produce better summaries (Böhm et al., 2019).”
- MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers (2021, background): “They capture task-specific properties; see the surveys [8, 48].”
- Scarecrow: A Framework for Scrutinizing Machine Text (2021, background): “Text is typically evaluated using a few global criteria, such as fluency and relevance, using discrete (e.g., 5-point) (Sai et al., 2020) or continuous scales (Novikova et al., 2018).”