Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generation

By Mingkai Deng, Bowen Tan, Zhengzhong Liu, Eric P. Xing, Zhiting Hu. 2021

83 citations recorded by the source corpus.

Natural language generation (NLG) spans a broad range of tasks, each of which serves for specific objectives and desires different properties of generated text. The complexity makes automatic evaluation of NLG particularly challenging. Previous work has typically focused on a single task and developed individual evaluation metrics based on specific intuitions. In this paper, we propose a unifying perspective based on the nature of information change in NLG tasks, including compression (e.g., summarization), transduction (e.g., text rewriting), and creation (e.g., dialog). _Information alignment_ between input, context, and output text plays a common central role in characterizing the generation. With automatic alignment prediction models, we develop a family of interpretable metrics that are suitable for evaluating key aspects of different NLG tasks, often without need of gold reference data. Experiments show the uniformly designed metrics achieve stronger or comparable correlations with human judgement compared to state-of-the-art metrics in each of diverse tasks, including text summarization, style transfer, and knowledge-grounded dialog.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “…(Bangalore et al., 2000), overlap between “basic elements” (Hovy et al., 2005),5 automatically constructed content units (Tauchmann and Mieskes, 2020) using the Pyramid framework by Nenkova and Passonneau (2004), dependency parses (Pratapa et al., 2021), or sequence alignment (Deng et al., 2021).”
Towards a Unified Multi-Dimensional Evaluator for Text Generation (2022, methodology): “For the score calculation, we follow previous work to compute sentence-level average scores for fluency and consistency (Laban et al., 2021) in summarization, and sentence-level cumulative scores for engagingness (Deng et al., 2021), while the rest is calculated as Equation 1.”
TRUE: Re-evaluating Factual Consistency Evaluation (2022, methodology): “CTC (Deng et al., 2021) measures the average token-level alignment of the generated text w.”
SMART: Sentences as Basic Units for Text Evaluation (2022, background): “…2022), where a summary is considered factual if all its facts are entailed by the source document, model-based approaches (Kryscinski et al., 2020; Deng et al., 2021), where a model is trained to detect factual errors in the summary, and question answering (QA) based approaches (Durmus et al.,…”
Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation (2022, methodology): “CTC (Deng et al., 2021) proposes metrics for Compression, transduction, and creation tasks as variations of textual alignment.”
RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning (2022, methodology): “Evaluation Details For automatic evaluation, We measure Content using the CTC metric (Deng et al., 2021) To evaluate Fluency, we rate output grammaticality using the classifier from Krishna et al. (2020).”
AlignScore: Evaluating Factual Consistency with A Unified Alignment Function (2023, methodology): “…consistency errors, such as contradictions with input information or hallucinations irrelevant to the 1Our code is available at https://github.com/ yuh-zha/AlignScore. context (Cao et al., 2018; Kryscinski et al., 2019; Nie et al., 2019a; Tan et al., 2020; Maynez et al., 2020; Deng et al., 2021).”
CTRLEval: An Unsupervised Reference-Free Metric for Evaluating Controlled Text Generation (2022, unknown): “However, controlled text generation tasks typically involve multiple evaluation aspects (Deng et al., 2021), including the quality of generated texts themselves and the relationship between generated texts and control variables.”