News Summarization and Evaluation in the Era of GPT-3

By Tanya Goyal, Junyi Jessy Li, Greg Durrett. 2022

502 citations recorded by the source corpus.

The recent success of prompting large language models like GPT-3 has led to a paradigm shift in NLP research. In this paper, we study its impact on text summarization, focusing on the classic benchmark domain of news summarization. First, we investigate how GPT-3 compares against fine-tuned models trained on large summarization datasets. We show that not only do humans overwhelmingly prefer GPT-3 summaries, prompted using only a task description, but these also do not suffer from common dataset-specific issues such as poor factuality. Next, we study what this means for evaluation, particularly the role of gold standard test sets. Our experiments show that both reference-based and reference-free automatic metrics cannot reliably evaluate GPT-3 summaries. Finally, we evaluate models on a setting beyond generic summarization, specifically keyword-based summarization, and show how dominant fine-tuning approaches compare to prompting. To support further research, we release: (a) a corpus of 10K generated summaries from fine-tuned and prompt-based models across 4 standard summarization benchmarks, (b) 1K human preference judgments comparing different systems for generic- and keyword-based summarization.

Subfields: Artificial Intelligence, Large Language Models, Summarization

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Percy Liang
  2. Mohit Bansal
  3. Luke S. Zettlemoyer
  4. Caiming Xiong
  5. Graham Neubig
  6. Jianfeng Gao
  7. Yejin Choi
  8. Noah A. Smith

In Large Language Models

  1. Dragomir R. Radev
  2. Daphne Ippolito
  3. Yixin Liu
  4. Percy Liang
  5. Florian Tramèr
  6. Nicholas Carlini
  7. Pengfei Liu
  8. Matthew Jagielski

In Summarization

  1. Yixin Liu
  2. Pengfei Liu
  3. Dragomir R. Radev
  4. Matei A. Zaharia
  5. James Y. Zou
  6. K. McKeown
  7. Tatsunori B. Hashimoto
  8. Lingjiao Chen

Citations in the corpus