Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategies

By Max Grusky, Mor Naaman, Yoav Artzi. 2018

629 citations recorded by the source corpus.

We present NEWSROOM, a summarization dataset of 1.3 million articles and summaries written by authors and editors in newsrooms of 38 major news publications. Extracted from search and social media metadata between 1998 and 2017, these high-quality summaries demonstrate high diversity of summarization styles. In particular, the summaries combine abstractive and extractive strategies, borrowing words and phrases from articles at varying rates. We analyze the extraction strategies used in NEWSROOM summaries against other datasets to quantify the diversity and difficulty of our new data, and train existing methods on the data to evaluate its utility and challenges. The dataset is available online at summari.es.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

The Natural Language Decathlon: Multitask Learning as Question Answering (2018, methodology): “While new corpora like NEWSROOM [Grusky et al., 2018] are even larger, CNN/DM remains the current standard benchmark, so we include it in decaNLP and encourage augmentation with datasets like NEWSROOM.”
CTRL: A Conditional Transformer Language Model for Controllable Generation (2019, background): “…1 , submissions from 45 subreddits, OpenWebText 2 , a large collection of news data (Hermann et al., 2015; Barrault et al., 2019; Sandhaus, 2008; Grusky et al., 2018), Amazon Reviews (McAuley et al., 2015), Europarl and UN data from WMT (En-De, En-Es, En-Fr) (Barrault et al., 2019),…”
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “articles, they typically follow best practices to provide salient information early on in the article (Grusky et al., 2018).”
Finetuned Language Models Are Zero-Shot Learners (2021, neutral): “Newsroom (Grusky et al., 2018) 6.”
MLSUM: The Multilingual Summarization Corpus (2020, methodology): “NEWSROOM Composed of 1.3M articles (Grusky et al., 2018), and featuring high diversity in terms of publishers, the summaries associated with English news articles were extracted from the Web pages metadata: they were originally written to be used in search engines and social media.”
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization (2019, methodology): “NEWSROOM (Grusky et al., 2018) is a large dataset containing 1.”
Don’t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization (2018, methodology): “Neural approaches to NLP and their ability to learn continuous features without recourse to pre-processing tools or linguistic annotations have driven the development of large-scale document summarization datasets (Sandhaus, 2008; Hermann et al., 2015; Grusky et al., 2018).”
Few-Shot Text Generation with Natural Language Instructions (2020, background): “• NEWSROOM (Grusky et al., 2018): Gener-ate summaries for articles from various major publications.”
Better Rewards Yield Better Summaries: Learning to Summarise Without References (2019, background): “The reliability of ROUGE as RL reward is further challenged by the fact that most large-scale summarisation datasets only have one reference summary available for each input document (e.g. CNN/DailyMail (Hermann et al., 2015; See et al., 2017) and NewsRooms (Grusky et al., 2018)).”
GPTScore: Evaluate as You Desire (2023, methodology): “We adopt SummEval (Bhan-dari et al., 2020), REALSumm (Bhandari et al., 2020), NEWSROOM (Grusky et al., 2018), and QAGS_XSUM (Wang et al., 2020b) datasets.”
Measuring Data (2022, background): “, 2020); “intrinsic metrics”, to quantify the properties of a dataset (Grusky et al., 2018; Bommasani and Cardie, 2020a); “statistical tendencies”, described as “present” in datasets (Meister and Cotterell, 2021a); “statistical properties” to surface issues with dataset contents (Paullada et al.”
Multi-News: A Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model (2019, methodology): “Grusky et al. (2018) additionally deﬁne three measures of the extractive nature of a dataset, which we use here for a comparison.”
Evaluating the Factual Consistency of Abstractive Text Summarization (2019, background): “…learning (Mc-Cann et al., 2017; Peters et al., 2018; Devlin et al., 2018), and availability of large-scale supervised datasets (Sandhaus, 2008; Nallapati et al., 2016; Grusky et al., 2018; Narayan et al., 2018; Sharma et al., 2019) allowed deep learning-based approaches to dominate the ﬁeld.”
Neural Text Summarization: A Critical Evaluation (2019, background): “…et al., 2015; Vinyals et al., 2015; Vaswani et al., 2017) and the availability of large scale data (Sandhaus, 2008; Nallapati et al., 2016a; Grusky et al., 2018) enabled the transition from systems based on expert knowledge and heuristics to data-driven approaches powered by end-to-end…”
Abstractive Summarization of Reddit Posts with Multi-level Memory Networks (2018, methodology): “With quantitative evaluation and user studies via AMT, we show that our model outperforms state-of-the-art abstractive summarization methods on both Reddit TIFU, News-room abstractive subset and XSum dataset.”
BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarization (2019, methodology): “…have several drawbacks such as limited training data (Document Understanding Conference 2 ), shorter summaries (Gigaword (Napoles et al., 2012), XSum (Narayan et al., 2018), and News-room (Grusky et al., 2018 news reporting, summary-worthy content is non-uniformly distributed within each article.”
MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance (2019, methodology): “, 2015) and Newsroom (Grusky et al., 2018) use a single reference summary, making it harder to obtain unbiased results when only lexical overlap is considered during summary evaluation.”
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization (2020, methodology): “Prior work measures abstractiveness by overlapped text spans between the summary and the document (Grusky et al., 2018; Zhang et al., 2018), or indirectly by the effectiveness of extractive baselines such as LEAD-3 (Nallapati et al.”
On Extractive and Abstractive Neural Document Summarization with Transformer Language Models (2020, methodology): “…different large-scale and long document summarization datasets - arXiv, PubMed (Cohan et al., 2018), bigPatent (Sharma et al., 2019) and Newsroom (Grusky et al., 2018a Evaluation We evaluate our method using full-length F-1 ROUGE scores (Lin, 2004) and re-used the code from (Cohan et al., 2018)…”
WikiHow: A Large Scale Text Summarization Dataset (2018, methodology): “Nallapati et al. (2017) has used the entity anonymization to create the anonymized version of the dataset while See et al. (2017) NEWSROOM: This corpus (Grusky et al., 2018) is the most recent large-scale dataset introduced for text summarization.”