SummScreen: A Dataset for Abstractive Screenplay Summarization

By Mingda Chen, Zewei Chu, Sam Wiseman, Kevin Gimpel. 2021

104 citations recorded by the source corpus.

We introduce SummScreen, a summarization dataset comprised of pairs of TV series transcripts and human written recaps. The dataset provides a challenging testbed for abstractive summarization for several reasons. Plot details are often expressed indirectly in character dialogues and may be scattered across the entirety of the transcript. These details must be found and integrated to form the succinct plot descriptions in the recaps. Also, TV scripts contain content that does not directly pertain to the central plot but rather serves to develop characters or provide comic relief. This information is rarely contained in recaps. Since characters are fundamental to TV series, we also propose two entity-centric evaluation metrics. Empirically, we characterize the dataset by evaluating several methods, including neural models and those based on nearest neighbors. An oracle extractive approach outperforms all benchmarked models according to automatic metrics, showing that the neural models are unable to fully exploit the input transcripts. Human evaluation and qualitative analysis reveal that our non-oracle models are competitive with their oracle counterparts in terms of generating faithful plot events and can benefit from better content selectors. Both oracle and non-oracle models generate unfaithful facts, suggesting future research directions.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

SCROLLS: Standardized CompaRison Over Long Language Sequences (2022, methodology): “7 19,402 SummScreenFD (Chen et al., 2021) Summ TV ROUGE 5,639 100.”
UL2: Unifying Language Learning Paradigms (2022, methodology): “We use the Scrolls benchmark (Shaham et al., 2022) which comprises of seven component tasks including GovReport (Huang et al., 2021), SumScr (Chen et al., 2021), QMSUm (Zhong et al., 2021), QASPER (Dasigi et al., 2021), NarrativeQA (Kočiský et al., 2018), QuaLITY (Pang et al., 2021), and…”
CONFIT: Toward Faithful Dialogue Summarization with Linguistically-Informed Contrastive Fine-tuning (2021, background): “…attentions (Mc-035 Cowan et al., 2005; Gliwa et al., 2019; Koay et al.,036 2020; Zhang et al., 2021; Zhong et al., 2021; Zhu037 et al., 2021; Chen et al., 2021a; Li et al., 2021; Chen038 et al., 2021c; Fabbri et al., 2021; Chen et al., 2021d).039 The goal of dialogue summarization is to…”
The NLP Task Effectiveness of Long-Range Transformers (2022, unknown): “(Chen et al., 2021; Huang et al., 2021), abstractive QA (aQA) (Dasigi et al., 2021; Pang et al., 2022).”
Unifying Language Learning Paradigms (2022, methodology): “We use the Scrolls benchmark (Shaham et al., 2022) which comprises of seven component tasks including GovReport (Huang et al., 2021), SumScr (Chen et al., 2021), QMSUm (Zhong et al., 2021), QASPER (Dasigi et al., 2021), NarrativeQA (Ko ˇ cisk ´ y et al., 2018), QuaLITY (Pang et al., 2021), and…”
SQuALITY: Building a Long-Document Summarization Dataset the Hard Way (2022, background): “SummScreen (Chen et al., 2022) consists of fan-written transcripts of TV”
Retentive Network: A Successor to Transformer for Large Language Models (2023, background): “(Zhong et al., 2021), GovReport (Huang et al., 2021), SummScreen (Chen et al., 2021; Shaham et al., 2022”
LongEmbed: Extending Embedding Models for Long Context Retrieval (2024, background): “This necessitates the ability to process and reason over long context, ensuring that answers cannot be obtained by merely focusing on a short span within the document.”