Benchmarking Large Language Models for News Summarization

By Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, K. McKeown, Tatsunori B. Hashimoto. Transactions of the Association for Computational Linguistics, 2023

820 citations recorded by the source corpus.

Large language models (LLMs) have shown promise for automatic summarization but the reasons behind their successes are poorly understood. By conducting a human evaluation on ten LLMs across different pretraining methods, prompts, and model scales, we make two important observations. First, we find instruction tuning, not model size, is the key to the LLM’s zero-shot summarization capability. Second, existing studies have been limited by low-quality references, leading to underestimates of human performance and lower few-shot and finetuning performance. To better evaluate LLMs, we perform human evaluation over high-quality summaries we collect from freelance writers. Despite major stylistic differences such as the amount of paraphrasing, we find that LLM summaries are judged to be on par with human written summaries.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/a4a41319d5805a29316f24ed9519f09db77d4c29

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment (2023, methodology): “We use the dataset collected in Zhang et al. (2023), where they first ask freelance writers to write high-quality summaries for news articles, and then ask annotators to compare human-written summaries and LLM-3 Figure 2: Averaged G-E VAL -4’s scores for human-written summaries and GPT-3.”
Hallucination of Multimodal Large Language Models: A Survey (2024, neutral): “Recently, the emergence of large language models (LLMs) [29, 81, 85, 99, 132] has dominated a wide range of tasks in natural language processing (NLP), achieving unprecedented progress in language understanding [39, 47], generation [128, 140] and reasoning [20, 58, 87, 107, 115].”
Large language models empowered agent-based modeling and simulation: a survey and perspectives (2023, positive): “…parameters and extensive pre-training corpus, LLMs have shown astonishing abilities not only in natural language processing tasks (Li et al., 2023a; Zhang et al., 2024c) such as text generation, summarization, translation, etc., but also in complex reasoning and planning tasks, such as solving…”
Towards Measuring the Representation of Subjective Global Opinions in Language Models (2023, neutral): “…decision-making processes such as detecting bias and toxicity (Hartvigsen et al., 2022), steering model outputs to abide by ethical principles (Bai et al., 2022b), generating model evaluations (Perez et al., 2022b), and summarizing the most important information from articles (Zhang et al., 2023).”
Trusting Your Evidence: Hallucinate Less with Context-aware Decoding (2023, methodology): “Following prior work (Zhang et al., 2023; Zhou et al., 2023), we evaluate the models using prompting.”
A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2023, neutral): “…language processing (NLP), achieving unprecedented progress in language understanding (Hendrycks et al., 2021; Huang et al., 2023c), generation (Zhang et al., 2023f; Zhu et al., 2023b) and reasoning (Wei et al., 2022; Kojima et al., 2022; Qiao et al., 2022; Yu et al., 2023a; Chu et al., 2023).”
Efficient Streaming Language Models with Attention Sinks (2023, background): “…language processing applications such as dialog systems (Schulman et al., 2022; Taori et al., 2023; Chiang et al., 2023), document summarization (Goyal & Durrett, 2020; Zhang et al., 2023a), code completion (Chen et al., 2021; Rozière et al., 2023) and question answering (Kamalloo et al., 2023).”
Benchmarking and Defending against Indirect Prompt Injection Attacks on Large Language Models (2023, neutral): “This demonstrates that current LLMs are not effectively resistant to indirect prompts and also highlights the dangers of indirect prompt injection attacks.”
H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models (2023, neutral): “Large Language Models (LLMs) have demonstrated remarkable proficiency in a wide range of natural language processing applications such as content creation, summarization, and dialogue systems [1, 2, 3, 4].”
A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation (2025, neutral): “Structured prompts (Experiments 3, 7, 8, and 4) .”
Markov-Enhanced Clustering for Long Document Summarization: Tackling the 'Lost in the Middle' Challenge with Large Language Models (2025, unknown): “Large language models (LLMs) [8] have excelled, particularly in abstractive summarization, and are driving significant advancements in this area [9].”