UNION: An Unreferenced Metric for Evaluating Open-ended Story Generation

By Jian Guan, Minlie Huang. 2020

81 citations recorded by the source corpus.

Despite the success of existing referenced metrics (e.g., BLEU and MoverScore), they correlate poorly with human judgments for open-ended text generation including story or dialog generation because of the notorious one-to-many issue: there are many plausible outputs for the same input, which may differ substantially in literal or semantics from the limited number of given references. To alleviate this issue, we propose UNION, a learnable unreferenced metric for evaluating open-ended story generation, which measures the quality of a generated story without any reference. Built on top of BERT, UNION is trained to distinguish human-written stories from negative samples and recover the perturbation in negative stories. We propose an approach of constructing negative samples by mimicking the errors commonly observed in existing NLG models, including repeated plots, conflicting logic, and long-range incoherence. Experiments on two story datasets demonstrate that UNION is a reliable measure for evaluating the quality of generated stories, which correlates better with human judgments and is more generalizable than existing state-of-the-art metrics.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Survey of Hallucination in Natural Language Generation (2022, background): “It is common for NLG tasks to have multiple plausible outputs from the same input, which is known as one-to-many mapping [64, 173].”
Scarecrow: A Framework for Scrutinizing Machine Text (2021, background): “…(Banerjee and Lavie, 2005 (Gu et al., 2020), which diffuses evaluation onto a multidimensional space and assigns a single holistic score; UNION (Guan and Huang, 2020), which learns to distinguish human-written stories from negative samples by generating perturbations of human written stories;…”
ZeroGen: Efficient Zero-shot Learning via Dataset Generation (2022, methodology): “Second, Z ERO G EN can serve as an unreferenced evaluation method for text generation (Guan and Huang, 2020; Pillutla et al., 2021): the downstream tasks’ performance is dominated by the quality of the synthesized text, thus can serve as an indirect measure of the generation models and algorithms.”
CTRLEval: An Unsupervised Reference-Free Metric for Evaluating Controlled Text Generation (2022, unknown): “We follow the existing work (Guan and Huang, 2020; Yuan et al., 2021) to adopt Pearson (r), Spearman (ρ), and Kendall (τ ) correlation coefficients between automatic metrics and human judgments”
MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers (2021, background): “Automatic metrics have been proposed for specific domains such as generation of dialogues [55], stories [21], and others [43].”