The price of debiasing automatic metrics in natural language evalaution

By Arun Tejasvi Chaganty, Stephen O. Mussmann, Percy Liang. 2018

135 citations recorded by the source corpus.

For evaluating generation systems, automatic metrics such as BLEU cost nothing to run but have been shown to correlate poorly with human judgment, leading to systematic bias against certain model improvements. On the other hand, averaging human judgments, the unbiased gold standard, is often too expensive. In this paper, we use control variates to combine automatic metrics with human evaluation to obtain an unbiased estimator with lower cost than human evaluation alone. In practice, however, we obtain only a 7-13% cost reduction on evaluating summarization and open-response question answering systems. We then prove that our estimator is optimal: there is no unbiased estimator with lower cost. Our theory further highlights the two fundamental bottlenecks—the automatic metric and the prompt shown to human evaluators—both of which need to be improved to obtain greater cost savings.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

CoQA: A Conversational Question Answering Challenge (2018, background): “Finding the perfect evaluation metric for abstractive responses is still a challenging problem (Liu et al., 2016; Chaganty et al., 2018) and beyond the scope of our work.”
Towards a Human-like Open-Domain Chatbot (2020, background): “h other automatic metrics in other tasks, e.g., perplexity vs. BLEU in translation (Luong et al., 2015). Another interesting line of work is to combine human evaluation with either automatic metrics (Chaganty et al., 2018) or with model likelihood (Hashimoto et al., 2019). While theoretically motiva”
Learning to summarize from human feedback (2020, background): “Indeed, existing automatic metrics for evaluating summary quality, such as ROUGE [39], have received criticism for poor correlation with human judgments [55, 45, 6, 33].”
Learning to Compare for Better Training and Evaluation of Open Domain Natural Language Generation Models (2020, background): “Another line of research on NLG evaluation is to unify human evaluation with statistical evaluation (Hashimoto et al. 2019; Chaganty et al. 2018).”
Better Rewards Yield Better Summaries: Learning to Summarise Without References (2019, methodology): “The human judgement scores we use are from Chaganty et al. (2018), collected as follows.”
From Recognition to Cognition: Visual Commonsense Reasoning (2018, methodology): “lit our full dataset (by movie) into 11 folds, each of which is individually run through Eq1. Two were pulled aside for validation and testing. 7Constructing new metrics is an active line of research [9,14]. 8We ﬁnetune P rel (BERT), on the annotated data (taking steps to avoid data leakage), wherea”
BERTScore: Evaluating Text Generation with BERT (2019, methodology): “equence. CIDER (Vedantam et al., 2015) is an image captioning metric that computes cosine similarity between tf–idf weighted n-grams. We adopt a similar approach to weigh tokens differently. Finally, Chaganty et al. (2018) and Hashimoto et al. (2019) combine automatic metrics with human judgments fo”
BLEURT: Learning Robust Metrics for Text Generation (2020, background): “Thus, they have been repeatedly shown to correlate poorly with human judgment, in particular when all the systems to compare have a similar level of accuracy (Liu et al., 2016; Novikova et al., 2017; Chaganty et al., 2018).”
Unifying Human and Statistical Evaluation for Natural Language Generation (2019, background): “, 2002) and ROUGE (Lin and Rey, 2004) capture quality better than perplexity but still correlate poorly with human evaluation and fail to capture diversity (Novikova et al., 2017; Chaganty et al., 2018).”
Handling Divergent Reference Texts when Evaluating Table-to-Text Generation (2019, background): “Chaganty et al. (2018) point out that hill-climbing on an automatic metric is meaningless if that metric has a low instance-level correlation to human judgments.”
MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance (2019, neutral): “Our ﬁnding agrees with the observations found in Chaganty et al. (2018); Novikova et al. (2017): lexical metrics correlate better with human judgments on texts of low quality than high quality.”
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization (2020, background): “The final evaluation should still rely on human annotation or human-in-the-loop methods (Chaganty et al., 2018).”
Challenges in Building Intelligent Open-domain Dialog Systems (2019, background): “[15] combined automatic metrics with human evaluation to obtain an unbiased estimator with lower cost than using solely human evaluation.”
GO FIGURE: A Meta Evaluation of Factuality in Summarization (2020, background): “The scoring function H(D,Si) represented by human evaluation is a gold standard for assessment of generation quality (Chaganty et al., 2018), so M(D,Si) should be an approximation.”
Exploring Content Selection in Summarization of Novel Chapters (2020, methodology): “Human Evaluation: Given questions about the reliability of ROUGE (Novikova et al., 2017; Chaganty et al., 2018), we perform human evaluation to assess which system is best at content selection.”
Best practices for the human evaluation of automatically generated text (2019, background): “We have provided an overview of the current state of human evaluation in NLG, and presented a set of best practices, summarized in Table 3.”
The statistical advantage of automatic NLG metrics at the system level (2021, methodology): “We wish to highlight several works which inspired the elements of ours: Chaganty et al. (2018) and Hashimoto et al. (2019) formalize metrics as statistical estimators and provide understanding of their statistical properties and limits.”
G ENIE A Leaderboard for Human-in-the-Loop Evaluation of Text Generation (2021, unknown): “(2018).”