Learning to Extract Coherent Summary via Deep Reinforcement Learning

By Yuxiang Wu, Baotian Hu. 2018

180 citations recorded by the source corpus.

Coherence plays a critical role in producing a high-quality summary from a document. In recent years, neural extractive summarization is becoming increasingly attractive. However, most of them ignore the coherence of summaries when extracting sentences. As an effort towards extracting coherent summaries, we propose a neural coherence model to capture the cross-sentence semantic and syntactic coherence patterns. The proposed neural coherence model obviates the need for feature engineering and can be trained in an end-to-end fashion using unlabeled data. Empirical results show that the proposed neural coherence model can efficiently capture the cross-sentence coherence patterns. Using the combined output of the neural coherence model and ROUGE package as the reward, we design a reinforcement learning method to train a proposed neural extractive summarizer which is named Reinforced Neural Extractive Summarization (RNES) model. The RNES model learns to optimize coherence and informative importance of the summary simultaneously. The experimental results show that the proposed RNES outperforms existing baselines and achieves state-of-the-art performance in term of ROUGE on CNN/Daily Mail dataset. The qualitative evaluation indicates that summaries produced by RNES are more coherent and readable.

Subfields: Artificial Intelligence, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Direct Preference Optimization: Your Language Model is Secretly a Reward Model (2023, methodology): “In particular, we formulate the following optimization problem max where � is a parameter controlling the deviation from the base reference policy ⇡ ref , namely the initial SFT model ⇡ SFT .”
Fine-Tuning Language Models from Human Preferences (2019, background): “…functions such as BLEU for translation (Ranzato et al., 2015; Wu et al., 2016), ROUGE for summarization (Ranzato et al., 2015; Paulus et al., 2017; Wu and Hu, 2018; Gao et al., 2019b), music theory-based rewards (Jaques et al., 2017), or event detectors for story generation (Tambwekar et al.,…”
Learning to summarize from human feedback (2020, background): “There is also a rich literature on using RL to optimize automatic metrics for NLP tasks, such as ROUGE for summarization [50, 65, 45, 15, 19], BLEU for translation [50, 66, 1, 43], and other domains [61, 27, 26].”
Evaluating the Factual Consistency of Abstractive Text Summarization (2019, background): “…2019; Zhang et al., 2019), attention and copying mechanisms (See et al., 2017; Cohan et al., 2018), and multi-objective training strategies (Guo et al., 2018; Pasunuru and Bansal, 2018), including reinforcement learning techniques (Kryściński et al., 2018; Dong et al., 2018; Wu and Hu, 2018).”
Neural Text Summarization: A Critical Evaluation (2019, methodology): “…Pasunuru and Bansal, 2018; Kry´sci´nski et al., 2018), reinforcement learning strategies (Paulus et al., 2017; Narayan et al., 2018b; Dong et al., 2018; Wu and Hu, 2018), and hybrid extractive-abstractive models (Liu et al., 2018; Hsu et al., 2018; Gehrmann et al., 2018; Chen and Bansal, 2018).”
BanditSum: Extractive Summarization as a Contextual Bandit (2018, methodology): “In future work, we will explore the direction of adding an extra coherence reward (Wu and Hu, 2018) to im-”
Guiding Extractive Summarization with Question-Answering Rewards (2019, methodology): “These studies build distributed sentence representations using neural networks (Cheng and Lapata, 2016; Yasunaga et al., 2017) and use reinforcement learning to optimize the evaluation metric (Narayan et al., 2018b) and improve summary coherence (Wu and Hu, 2018).”
RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (2023, unknown): “While initial applications of RL on tasks such as translation (Wu et al., 2016, 2018) and summarization (Gao et al., 2019; Wu and Hu, 2018) used automatic evaluation metrics as rewards, such simplified formulations of rewards did not fully align with human notions of quality.”
RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (2023, methodology): “While initial applications of RL on tasks such as translation (Wu et al., 2016; 2018) and summarization (Gao et al., 2019; Wu & Hu, 2018) used automatic evaluation metrics as rewards, such simpliﬁed formulations of rewards did not fully align with human notions of quality.”
Searching for Effective Neural Extractive Summarization: What Works and What’s Next (2019, methodology): “In addition, some work uses reinforcement learning technique (Narayan et al., 2018; Wu and Hu, 2018; Chen and Bansal, 2018), which can provide more direct optimization goals.”
Reward Learning for Efficient Reinforcement Learning in Extractive Document Summarisation (2019, methodology): “Encoding document-dependent features requires more sophisticated neural models [ Wu and Hu, 2018; Narayan et al. , 2018 ] which require considerable time, data and parameter tuning.”
Offline RL for Natural Language Generation with Implicit Language Q Learning (2022, background): “A number of prior works have explored combining online RL methods with language models for natural language tasks such as machine translation or summarization [19, 20, 21, 22, 23, 24, 25, 26, 27, 28].”
An Overview of Natural Language State Representation for Reinforcement Learning (2020, methodology): “Wu & Hu (2018) built a document encoding with a CNN operating at word level and a BiGRU at sentence level, whereas Narayan et al. (2018) and Chen & Bansal (2018) used a CNN at sentence level and an LSTM or BiLSTM at document level to capture global information.”