Abstractive Summarization of Reddit Posts with Multi-level Memory Networks

By Byeongchang Kim, Hyunwoo Kim, Gunhee Kim. 2018

205 citations recorded by the source corpus.

We address the problem of abstractive summarization in two directions: proposing a novel dataset and a new model. First, we collect Reddit TIFU dataset, consisting of 120K posts from the online discussion forum Reddit. We use such informal crowd-generated posts as text source, in contrast with existing datasets that mostly use formal documents as source such as news articles. Thus, our dataset could less suffer from some biases that key sentences usually located at the beginning of the text and favorable summary candidates are already inside the text in similar forms. Second, we propose a novel abstractive summarization model named multi-level memory networks (MMN), equipped with multi-level memory to store the information of text from different levels of abstraction. With quantitative evaluation and user studies via Amazon Mechanical Turk, we show the Reddit TIFU dataset is highly abstractive and the MMN outperforms the state-of-the-art summarization models.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus (2021, methodology): “We examine contamination of target text in test sets of datasets for three generation tasks: (i) abstractive summarization (TIFU, Kim et al., 2019; XSum, Narayan et al., 2018), (ii) table-to-text generation (WikiBio, Lebret et al., 2016), and (iii) graph-to-text generation (AMR-to-text, LDC2017T10).”
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization (2019, background): “On WikiHow and Reddit TIFU, the best configurations were Unigram 128k and 64k respectively.”
Few-Shot Text Generation with Natural Language Instructions (2020, background): “• Reddit TIFU (Kim et al., 2019): Generate summaries for posts from the TIFU community in Reddit.”
CM3: A Causal Masked Multimodal Model of the Internet (2022, background): “Reddit TIFU (Kim et al., 2018) contains summaries of Reddit posts.”
CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP (2021, background): “…Lai et al. 2017 race-middle qa/multiple-choice qa Lai et al. 2017 reddit_tifu-title cg/summarization Kim et al. 2019 reddit_tifu-tldr cg/summarization Kim et al. 2019 ropes qa/machine reading comprehension Lin et al. 2019 rotten_tomatoes cls/sentiment analysis Pang and Lee 2005 samsum…”
Better Fine-Tuning by Reducing Representational Collapse (2020, background): “While prior work in non-standard ﬁnetuning methods tends to focus on sentence prediction and GLUE tasks (Jiang et al., 2019; Zhu et al., 2019; Zhang et al., 2020), we look to improve abstractive summarization, due to its additional complexity and computational cost, speciﬁcally we look at three data”
Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation (2020, neutral): “, 2018), Reddit_tifu (Reddit) (Kim et al., 2019), and BigPatent (Sharma et al.”
Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning (2020, background): “They show improved performance on a range of natural language understanding and generation tasks including DailyMail/CNN (Hermann et al., 2015), Gigaword (Napoles et al., 2012), Reddit TIFU (Kim et al., 2019), and the GLUE benchmark.”
MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge (2022, methodology): “Just as humans frequently learn from the internet, agents should also be able to harvest practical knowledge encoded in large amounts of video demos [35, 67], multimedia tutorials [69], and forum discussions [112, 56, 47].”
Efficient Adaptation of Pretrained Transformers for Abstractive Summarization (2019, methodology): “We also compare against the Multi-Level Memory Network (MMN) [9] mentioned earlier.”
How well do you know your summarization datasets? (2021, unknown): “, 2018a), and many more (Wang and Ling, 2016; Koupaee and Wang, 2018; Kim et al., 2019; Ganesan et al., 2010) were collected by scraping a large collection of web-pages.”
G ENIE A Leaderboard for Human-in-the-Loop Evaluation of Text Generation (2021, methodology): “These evaluations are much larger than what was previously done in the ﬁeld, e.g., 100 instances for MT in Ma et al. (2018) or around 100 instances for summarization in Kim et al. (2019) and Hardy et al. (2019).”
AdaptSum: Towards Low-Resource Domain Adaptation for Abstractive Summarization (2021, background): “Social Media Kim et al. (2019) introduced an abstractive summarization dataset of Reddit TIFU posts, where the summary for each post come from its title.”
GSum: A General Framework for Guided Neural Abstractive Summarization (2020, methodology): “Reddit (Kim et al., 2019) is a highly abstractive dataset and we use its TIFU-long version.”
Generating Abstractive Summaries with Finetuned Language Models (2019, background): “To avoid this problem, Kim et al. (2018) and V¨olske et al. (2017) created reddit-based corpora with more abstractive target summaries that enable the evaluation of supervised models instead.”