Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation

By Yixin Liu, Alexander R. Fabbri, Pengfei Liu, Yilun Zhao, Linyong Nan, Ruilin Han, Simeng Han, Shafiq R. Joty, Chien-Sheng Wu, Caiming Xiong and 1 more. 2022

179 citations recorded by the source corpus.

Human evaluation is the foundation upon which the evaluation of both summarization systems and automatic metrics rests. However, existing human evaluation studies for summarization either exhibit a low inter-annotator agreement or have insufficient scale, and an in-depth analysis of human evaluation is lacking. Therefore, we address the shortcomings of existing summarization evaluation along the following axes: (1) We propose a modified summarization salience protocol, Atomic Content Units (ACUs), which is based on fine-grained semantic units and allows for a high inter-annotator agreement. (2) We curate the Robust Summarization Evaluation (RoSE) benchmark, a large human evaluation dataset consisting of 22,000 summary-level annotations over 28 top-performing systems on three datasets. (3) We conduct a comparative study of four human evaluation protocols, underscoring potential confounding factors in evaluation setups. (4) We evaluate 50 automatic metrics and their variants using the collected human annotations across evaluation protocols and demonstrate how our benchmark leads to more statistically stable and significant results. The metrics we benchmarked include recent methods based on large language models (LLMs), GPTScore and G-Eval. Furthermore, our findings have important implications for evaluating LLMs, as we show that LLMs adjusted by human feedback (e.g., GPT-3.5) may overfit unconstrained human evaluation, which is affected by the annotators’ prior, input-agnostic preferences, calling for more robust, targeted evaluation methods.

Subfields: Artificial Intelligence, Molecular Biology, Evaluation & Benchmarks, Summarization

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Graham Neubig
  2. Percy Liang
  3. Mohit Bansal
  4. Luke S. Zettlemoyer
  5. Hannaneh Hajishirzi
  6. Jianfeng Gao
  7. Noah A. Smith
  8. Yejin Choi

In Molecular Biology

  1. Percy Liang
  2. Hannaneh Hajishirzi
  3. Tianyi Zhang
  4. Wen-tau Yih
  5. Pengfei Liu
  6. K. McKeown
  7. Graham Neubig
  8. See-Kiong Ng

In Evaluation & Benchmarks

  1. Pengfei Liu
  2. Hannaneh Hajishirzi
  3. Graham Neubig
  4. Jinlan Fu
  5. Luke S. Zettlemoyer
  6. See-Kiong Ng
  7. Percy Liang
  8. Sewon Min

In Summarization

  1. Pengfei Liu
  2. Graham Neubig
  3. Weizhe Yuan
  4. K. McKeown
  5. Tatsunori B. Hashimoto
  6. Hannaneh Hajishirzi
  7. Jinlan Fu
  8. Esin Durmus

Citations in the corpus