MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers

By Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, Zaïd Harchaoui. 2021

548 citations recorded by the source corpus.

As major progress is made in open-ended text generation, measuring how close machine-generated text is to human language remains a critical open problem. We introduce MAUVE, a comparison measure for open-ended text generation, which directly compares the learnt distribution from a text generation model to the distribution of human-written text using divergence frontiers. MAUVE scales up to modern text generation models by computing information divergences in a quantized embedding space. Through an extensive empirical study on three open-ended generation tasks, we find that MAUVE identifies known properties of generated text, scales naturally with model size, and correlates with human judgments, with fewer restrictions than existing distributional evaluation metrics.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “noted by Pitler et al. (2010) who find correlations between some evaluation categories.”
Quark: Controllable Text Generation with Reinforced Unlearning (2022, background): “For generation quality, we report sequence-level repetition, defined as the proportion of repeated n-grams (rep-n), diversity (diverse) as measured by a fusion of different n-gram levels, and MAUVE [55], an automatic measure of how much the generated text distribution diverges from that of human-wri”
Contrastive Decoding: Open-ended Text Generation as Optimization (2022, methodology): “MAUVE (Pillutla et al., 2021) score (the higher the better) measures the distribution similarity between the set of generated text and the set of gold reference.”
RankGen: Improving Text Generation with Large Ranking Models (2022, methodology): “Table 3: A comparison between RANKGEN variants and baseline decoding algorithms using MAUVE (Pillutla et al., 2021), an automatic text generation metric with high human correlation.”
PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization (2023, background): “The previous qualitative analysis can be achieved either through human evaluators or through APIs of advanced language models, which is different from our motivation.”
Data Feedback Loops: Model-driven Amplification of Dataset Biases (2022, background): “, 2022), defined as the average similarity between prompts and corresponding model completions featurized by a sentence embedding model; 2) mauve score (Pillutla et al., 2021), defined as the difference in distributions between model-completed sentences and ground truth sentences, featurized by GPT-”
Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection (2023, methodology): “On the contrary, a larger weight results in lower MAUVE scores: when generation gets longer and more fluent, there are often more claims that are not fully supported by citations, consistent with findings by Liu et al. (2023a).”
Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads (2024, background): “Recent studies have revealed that direct sampling from a language model may lead to incoherent or nonsensical results (Pillutla et al., 2021; Holtzman et al., 2020).”
Enabling Large Language Models to Generate Text with Citations (2023, methodology): “We use MAUVE (Pillutla et al., 2021) to evaluate the fluency of the output (§C).”
A Contrastive Framework for Neural Text Generation (2022, background): “MAUVE [17] is a metric that measures the token distribution closeness between the generated text and human-written text.”
Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution (2023, methodology): “As most language metrics are meant for comparing conditional generations (Pillutla et al., 2021), we instead measure the generative perplexity of sampled sequences (using a GPT-2 large model for evaluation).”
Language Models Can See: Plugging Visual Controls in Text Generation (2022, methodology): “The hyperparameters of different methods are selected based on their optimal MAUVE [57] (detailed in Section §5.”
Continuous diffusion for categorical data (2022, methodology): “For experiments with a larger mask-conditional model, we also report the MAUVE metric (Pillutla et al., 2021), which is speciﬁcally designed for open-ended text generation and has been shown to correlate with human judgement (see §6.5).”
Typical Decoding for Natural Language Generation (2022, methodology): “As automatic metrics, we evaluate the generated text’s perplexity, Mauve score (Pillutla et al., 2021) with the reference text—a metric that uses LM embeddings to measure the similarity between two text distributions—as well as rep (Welleck et al.”
Training Trajectories of Language Models Across Scales (2022, methodology): “To have a better understanding of the overall quality of the generated sequences, we evaluate these sequences decoded with each configuration in Figure 6 using MAUVE scores (Pillutla et al., 2021).”
Factuality Enhanced Language Models for Open-Ended Text Generation (2022, background): “Evaluating open-ended text generation can be challenging due to the lack of ground-truth references for generation [29, 60].”
ZeroGen: Efficient Zero-shot Learning via Dataset Generation (2022, methodology): “Second, Z ERO G EN can serve as an unreferenced evaluation method for text generation (Guan and Huang, 2020; Pillutla et al., 2021): the downstream tasks’ performance is dominated by the quality of the synthesized text, thus can serve as an indirect measure of the generation models and algorithms.”
RAS: Retrieval-And-Structuring for Knowledge-Intensive LLM Generation (2025, unknown): “…ELI5, we use multiple metrics 6 : • ROUGE-L score to measure the longest common subsequence between prediction and reference (Lin, 2004) • MAUVE score (Pillutla et al., 2021) For Answer Generation (Figure 22), we design a prompt that focuses on synthesizing precise answers from structured…”