News Summarization and Evaluation in the Era of GPT-3
By Tanya Goyal, Junyi Jessy Li, Greg Durrett. 2022
502 citations recorded by the source corpus.
The recent success of prompting large language models like GPT-3 has led to a paradigm shift in NLP research. In this paper, we study its impact on text summarization, focusing on the classic benchmark domain of news summarization. First, we investigate how GPT-3 compares against fine-tuned models trained on large summarization datasets. We show that not only do humans overwhelmingly prefer GPT-3 summaries, prompted using only a task description, but these also do not suffer from common dataset-specific issues such as poor factuality. Next, we study what this means for evaluation, particularly the role of gold standard test sets. Our experiments show that both reference-based and reference-free automatic metrics cannot reliably evaluate GPT-3 summaries. Finally, we evaluate models on a setting beyond generic summarization, specifically keyword-based summarization, and show how dominant fine-tuning approaches compare to prompting. To support further research, we release: (a) a corpus of 10K generated summaries from fine-tuned and prompt-based models across 4 standard summarization benchmarks, (b) 1K human preference judgments comparing different systems for generic- and keyword-based summarization.
Subfields: Artificial Intelligence, Large Language Models, Summarization
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Percy Liang
- Mohit Bansal
- Luke S. Zettlemoyer
- Caiming Xiong
- Graham Neubig
- Jianfeng Gao
- Yejin Choi
- Noah A. Smith
In Large Language Models
- Dragomir R. Radev
- Daphne Ippolito
- Yixin Liu
- Percy Liang
- Florian Tramèr
- Nicholas Carlini
- Pengfei Liu
- Matthew Jagielski
In Summarization
- Yixin Liu
- Pengfei Liu
- Dragomir R. Radev
- Matei A. Zaharia
- James Y. Zou
- K. McKeown
- Tatsunori B. Hashimoto
- Lingjiao Chen
Citations in the corpus
- Are aligned neural networks adversarially aligned? (2023, methodology): “For instance, language models are commonly used to perform tasks like question answering, translation, and summarization [Chowdhery et al., 2022, Rae et al., 2022, Anil et al., 2023, Liang et al., 2022, Goyal et al., 2022].”
- How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation (2023, background): “This few-shot paradigm has demonstrated strong performance across multiple natural language processing (NLP) tasks (Ouyang et al., 2022; Goyal et al., 2022; Wei et al., 2022; Chowdhery et al., 2022).”
- SLiC-HF: Sequence Likelihood Calibration with Human Feedback (2023, background): “For summarization in particular, multiple studies have shown that summaries generated by models tuned with RLHF are preferred over the reference summaries in commonly used datasets [18, 10, 8].”
- BloombergGPT: A Large Language Model for Finance (2023, methodology): “, 2022) and the second is to perform extrinsic and task-specific evaluations by integrating them into user workflows (e.g., Lee et al., 2022b; Goyal et al., 2022).”
- Large Language Models for Software Engineering: Survey and Open Problems (2023, background): “As a result, they have been found capable of performing a wide range of language-related tasks, including text generation [15], answering questions [16], translation [17], summarization [18], and sentiment analysis [19].”
- Benchmarking and Defending against Indirect Prompt Injection Attacks on Large Language Models (2023, unknown): “This demonstrates that current LLMs are not effectively resistant to indirect prompts and also highlights the dangers of indirect prompt injection attacks.”
- How is ChatGPT's behavior changing over time? (2023, methodology): “Researchers have evaluated temporal shifts of previous (non-LLM) AI models (L. Chen et al., 2021).”
- Can Large Language Models Transform Computational Social Science? (2023, background): “It is well-known that automatic performance metrics fail to capture human preferences (Goyal et al., 2022; Liang et al., 2022).”
- A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity (2023, unknown): “Average of performances for CNN and DM from Goyal et al. (2022).”
- Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond (2023, unknown): “For summarization tasks, although LLMs do not have an obvious advantage over fine-tuned models under traditional automatic evaluation metrics, such as ROUGE [60], human evaluation results indicate that humans tend to prefer the results generated by LLMs [38, 127] compared to that of fine-tuned model”
- Benchmarking Large Language Models for News Summarization (2023, background): “On XSUM, the metrics have very low correlation with faithfulness and relevance but it is also because the reference summaries are terrible in these aspects (Table 3; also see Maynez et al., 2020).”
- A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets (2023, result): “This is consistent with the recent findings (Liu et al., 2023c; Goyal et al., 2022), where summaries from GPT-3.”
- Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation (2022, methodology): “Therefore, we join the previous work (Sun et al., 2019; Song et al., 2021; Gehrmann et al., 2022; Goyal et al., 2022) in advocating treating summary lengths as a separate aspect of summary quality in evaluation, as in earlier work in summarization research.”