Fine-tune BERT for Extractive Summarization

By Yang Liu. 2019

535 citations recorded by the source corpus.

BERT, a pre-trained Transformer model, has achieved ground-breaking performance on multiple NLP tasks. In this paper, we describe BERTSUM, a simple variant of BERT, for extractive summarization. Our system is the state of the art on the CNN/Dailymail dataset, outperforming the previous best-performed system by 1.65 on ROUGE-L. The codes to reproduce our results are available at this https URL

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, methodology): “Furthermore, while CNN/Daily Mail is posed as an abstractive summarization benchmark, purely extractive approaches have been shown to work well [Liu, 2019].”
BERTScore: Evaluating Text Generation with BERT (2019, background): “…inference, sentiment analysis, paraphrase detection, question answering, named entity recognition (Devlin et al., 2019), text summarization (Liu, 2019), contextual emotion detection (Huang et al., 2019), citation recommendation (Jeong et al., 2019), and document retrieval (Yang et al., 2019).”
Revisiting Few-sample BERT Fine-tuning (2020, background): “1 Introduction Fine-tuning self-supervised pre-trained models has significantly boosted state-of-the-art performance on Natural Language Processing (NLP) tasks [13, 28, 43, 51, 56].”
Evaluating the Factual Consistency of Abstractive Text Summarization (2019, background): “of-the-art solutions utilize self-attentive Transformer blocks (Liu, 2019; Liu and Lapata, 2019; Zhang et al., 2019), attention and copying mechanisms (See et al.”
Open Sesame: Getting inside BERT’s Linguistic Knowledge (2019, background): “jacency training objective. Simply using BERT’s representations in place of traditional embeddings has resulted in state-of-the-art performance on a range of downstream tasks including summarization (Liu, 2019), question answering and textual entailment (Devlin et al.,2018). It is still, however, un”
Mind The Facts: Knowledge-Boosted Coherent Abstractive Text Summarization (2020, methodology): “ation over the past years [11, 18]. Current state-of-the-art model for the extractive approach ﬁne-tunes a simple variant of the popular language model BERT [12] for the extractive summarization task [10]. On the other hand, abstractive approaches generate novel text, and are able to paraphrase sent”
Exploiting BERT for End-to-End Aspect-based Sentiment Analysis (2019, methodology): “Since directly applying RNN on the output of transformer, namely, the BERT representation h Lt , may lead to unstable training (Chen et al., 2018; Liu, 2019), we add additional layer-normalization (Ba et al., 2016), denoted as L N , when calculating the gates.”
Abstractive Summarization with Combination of Pre-trained Sequence-to-Sequence and Saliency Models (2020, methodology): “Pre-trained Language Models for Abstractive Summarization Liu (2019) used BERT for their sentence-level extractive summarization model.”
How well do you know your summarization datasets? (2021, unknown): “The past few years have witnessed major breakthroughs and improvements in automatic summarization (See et al., 2017; Celikyilmaz et al., 2018; Jadhav and Rajan, 2018; Liu and Lapata, 2019; Liu, 2019; Dou et al., 2020; Yuan et al., 2021; Liu et al., 2021).”
Extractive Summarization as Text Matching (2020, methodology): “Unlike the modified BERT used in (Liu, 2019; Bae et al., 2019), we directly use the original BERT to derive the semantically meaningful embeddings from document D and candidate summary C since we need not obtain the sentence-level representation.”
Less training, more repairing please: revisiting automated program repair via zero-shot learning (2022, unknown): “tasks such as text summarization [46], text classification [77], as well as question and response text generation [15].”
Encoder-Decoder Models Can Benefit from Pre-trained Masked Language Models in Grammatical Error Correction (2020, methodology): “MLMs are generally used in downstream tasks by ﬁne-tuning (Liu, 2019; Zhang et al., 2019), however, Zhu et al. (2020) demonstrated that it is more effective to provide the output of the ﬁnal layer of a MLM to the EncDec model as contextual embed-dings.”
Language model compression with weighted low-rank factorization (2022, background): “Language models built with transformers (Devlin et al., 2018) have attained extensive success in natural language tasks such as language modeling (Radford et al., 2018), text classiﬁcation (Wang et al., 2018), question answering (Rajpurkar et al., 2016), and summarization (Liu, 2019).”
DialogSum: A Real-Life Scenario Dialogue Summarization Dataset (2021, methodology): “Following previous summarization work (Liu, 2019; Pilault et al., 2020), we report R1, R2 and RL here.”
Think before you speak: Training Language Models With Pause Tokens (2023, background): “The idea of using tokens that occur only as an input has found its use in various forms, most commonly as <cls> (Chang et al., 2023; Liu, 2019; Devlin et al., 2019), <sep> or <mask> in BERT (Devlin et al.”