Unifying Human and Statistical Evaluation for Natural Language Generation

By Tatsunori B. Hashimoto, Hugh Zhang, Percy Liang. 2019

250 citations recorded by the source corpus.

How can we measure whether a natural language generation system produces both high quality and diverse outputs? Human evaluation captures quality but not diversity, as it does not catch models that simply plagiarize from the training set. On the other hand, statistical evaluation (i.e., perplexity) captures diversity but not quality, as models that occasionally emit low quality samples would be insufficiently penalized. In this paper, we propose a unified framework which evaluates both diversity and quality, based on the optimal error rate of predicting whether a sentence is human- or machine-generated. We demonstrate that this error rate can be efficiently estimated by combining human and statistical evaluation, using an evaluation metric which we call HUSE. On summarization and chit-chat dialogue, we show that (i) HUSE detects diversity defects which fool pure human evaluation and that (ii) techniques such as annealing for improving quality actually decrease HUSE due to decreased diversity.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

GLTR: Statistical Detection and Visualization of Generated Text (2019, background): “This approach can be contextualized in the evaluation framework proposed by Hashimoto et al. (2019) who ﬁnd that human-written and generated text can be discriminated based on the model likelihood if the human acceptability is high.”
Defending Against Neural Fake News (2019, background): “…such as beam search work well for closed-ended generation tasks where the output contains the same information as the context (like machine translation), these approaches have been shown to produce degenerate text during open-ended generation (Hashimoto et al., 2019; Holtzman et al., 2019).”
The Curious Case of Neural Text Degeneration (2019, methodology): “However, recent analysis has shown that, while lowering the temperature improves generation quality, it comes at the cost of decreasing diversity (Caccia et al., 2018; Hashimoto et al., 2019).”
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “Hashimoto et al. (2019) used the same foundation to combine human and auto-”
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, background): “As argued by Hashimoto et al. (2019) among many others, NLG models intrinsically trade off diversity and quality.”
Towards a Human-like Open-Domain Chatbot (2020, background): “Another interesting line of work is to combine human evaluation with either automatic metrics (Chaganty et al., 2018) or with model likelihood (Hashimoto et al., 2019).”
All That’s ‘Human’ Is Not Gold: Evaluating Human Evaluation of Generated Text (2021, background): “A subfield of NLG analyzes the role of human evaluations, including discussions of the tradeoffs of human and automatic evaluations (Belz and Reiter, 2006; Hashimoto et al., 2019).”
Learning to Compare for Better Training and Evaluation of Open Domain Natural Language Generation Models (2020, background): “Another line of research on NLG evaluation is to unify human evaluation with statistical evaluation (Hashimoto et al. 2019; Chaganty et al. 2018).”
Social Bias Frames: Reasoning about Social and Power Implications of Language (2019, methodology): “We use BLEU2 and RougeL (F1) scores to capture word overlap between the generated inference and the references, which captures quality of generation (Galley et al., 2015; Hashimoto et al., 2019).”
Approximating Interactive Human Evaluation with Self-Play for Open-Domain Dialog Systems (2019, background): “Several authors have proposed training automatic predictors of human judgment or to combine human judgment with automatic metrics [17–19].”
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization (2020, background): “Recently, more work has focused on model-based evaluation using discriminators (Lowe et al., 2017; Hashimoto et al., 2019), entailment models (Falke et al., 2019a), information extraction (Wiseman et al., 2017; Goodrich et al., 2019), and question answering (Chen et al., 2018; Eyal et al., 2019).”
Challenges in Building Intelligent Open-domain Dialog Systems (2019, background): “[45] argued that human evaluation captures quality but not diversity while statistical evaluation (i.”
Evaluation of Text Generation: A Survey (2020, background): “Furthermore, even with a large group of annotators, there are some dimensions of generated text that are not well-suited to human evaluations, such as diversity (Hashimoto et al., 2019).”
With Little Power Comes Great Responsibility (2020, methodology): “…mixed effect regression), and compute power as a function of mean difference and sample 12We use publicly available or author-provided data from Hashimoto et al. (2019); Dathathri et al. (2020); Holtzman et al. (2020), and WMT19 (links in Appendix H.2). size.13 The resulting power estimates…”
HYPE: A Benchmark for Human eYe Perceptual Evaluation of Generative Models (2019, background): “Similar to related work in evaluating text generation [21], we suggest that diversity can be incorporated using the automated recall score measures diversity independently from precision F1/8 [51].”
What Context Features Can Transformer Language Models Use? (2021, background): “Recent work has shown that other metrics, such as diversity of outputs, are important for evaluating the quality of LMs as models for language generation (Hashimoto et al., 2019; Caccia et al., 2020).”
GO FIGURE: A Meta Evaluation of Factuality in Summarization (2020, background): “…for measuring quality of generated text fail to capture structural aspects of language like negation and poorly correlate with human judgements (Hashimoto et al., 2019; Clark et al., 2019; Sellam et al., 2020), leading to a rapidly progressing search for factuality-driven summarization metrics.”
CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities (2022, background): “Any LM’s performance fuctuates signifcantly depending on the preceding text [38], decoding parameters [27], among other factors.”
MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers (2021, methodology): “HUSE [24] aims to combine human judgements of Type I errors with Type II errors measured using perplexity under Q. Due to the costs of human evaluation, we consider HUSE, as well other metrics requiring human evaluation, such as single-pair evaluation, as complementary to MAUVE, which is an automati”