Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation
By Yixin Liu, Alexander R. Fabbri, Pengfei Liu, Yilun Zhao, Linyong Nan, Ruilin Han, Simeng Han, Shafiq R. Joty, Chien-Sheng Wu, Caiming Xiong and 1 more. 2022
179 citations recorded by the source corpus.
Human evaluation is the foundation upon which the evaluation of both summarization systems and automatic metrics rests. However, existing human evaluation studies for summarization either exhibit a low inter-annotator agreement or have insufficient scale, and an in-depth analysis of human evaluation is lacking. Therefore, we address the shortcomings of existing summarization evaluation along the following axes: (1) We propose a modified summarization salience protocol, Atomic Content Units (ACUs), which is based on fine-grained semantic units and allows for a high inter-annotator agreement. (2) We curate the Robust Summarization Evaluation (RoSE) benchmark, a large human evaluation dataset consisting of 22,000 summary-level annotations over 28 top-performing systems on three datasets. (3) We conduct a comparative study of four human evaluation protocols, underscoring potential confounding factors in evaluation setups. (4) We evaluate 50 automatic metrics and their variants using the collected human annotations across evaluation protocols and demonstrate how our benchmark leads to more statistically stable and significant results. The metrics we benchmarked include recent methods based on large language models (LLMs), GPTScore and G-Eval. Furthermore, our findings have important implications for evaluating LLMs, as we show that LLMs adjusted by human feedback (e.g., GPT-3.5) may overfit unconstrained human evaluation, which is affected by the annotators’ prior, input-agnostic preferences, calling for more robust, targeted evaluation methods.
Subfields: Artificial Intelligence, Molecular Biology, Evaluation & Benchmarks, Summarization
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Graham Neubig
- Percy Liang
- Mohit Bansal
- Luke S. Zettlemoyer
- Hannaneh Hajishirzi
- Jianfeng Gao
- Noah A. Smith
- Yejin Choi
In Molecular Biology
- Percy Liang
- Hannaneh Hajishirzi
- Tianyi Zhang
- Wen-tau Yih
- Pengfei Liu
- K. McKeown
- Graham Neubig
- See-Kiong Ng
In Evaluation & Benchmarks
- Pengfei Liu
- Hannaneh Hajishirzi
- Graham Neubig
- Jinlan Fu
- Luke S. Zettlemoyer
- See-Kiong Ng
- Percy Liang
- Sewon Min
In Summarization
- Pengfei Liu
- Graham Neubig
- Weizhe Yuan
- K. McKeown
- Tatsunori B. Hashimoto
- Hannaneh Hajishirzi
- Jinlan Fu
- Esin Durmus
Citations in the corpus
- GPTScore: Evaluate as You Desire (2023, methodology): “| In this paper, the criteria for choosing GPTS CORE variants are mainly designed to align the protocol of human judgments (Liu et al., 2022) that are used to evaluate the reliability of automated metrics.”
- Benchmarking Large Language Models for News Summarization (2023, background): “…(LLMs) have shown promising results in zero-/few-shot tasks across a wide range of domains (Chowdhery et al., 2022; Bai et al., 2022; Brown et al., 2020; Zhang et al., 2022) and raised significant interest for their potential for automatic summarization (Goyal et al., 2022; Liu et al., 2022a).”
- FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation (2023, background): “Computing FACTSCORE involves (1) breaking a generation into a series of atomic facts—short statements that each contain one piece of information (Nenkova and Passonneau, 2004; Shapira et al., 2019; Zhang and Bansal, 2021; Liu et al., 2022), and (2) assigning a binary label”
- A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets (2023, result): “This is consistent with the recent findings (Liu et al., 2023d; Goyal et al., 2022), where summaries from GPT-3.”
- FacTool: Factuality Detection in Generative AI - A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios (2023, methodology): “T OOL on RoSE (Liu et al., 2022).”