Reward Learning for Efficient Reinforcement Learning in Extractive Document Summarisation

By Yang Gao, Christian M. Meyer, Mohsen Mesgar, Iryna Gurevych. 2019

27 citations recorded by the source corpus.

Document summarisation can be formulated as a sequential decision-making problem, which can be solved by Reinforcement Learning (RL) algorithms. The predominant RL paradigm for summarisation learns a cross-input policy, which requires considerable time, data and parameter tuning due to the huge search spaces and the delayed rewards. Learning input-specific RL policies is a more efficient alternative, but so far depends on handcrafted rewards, which are difficult to design and yield poor performance. We propose RELIS, a novel RL paradigm that learns a reward function with Learning-to-Rank (L2R) algorithms at training time and uses this reward function to train an input-specific RL policy at test time. We prove that RELIS guarantees to generate near-optimal summaries with appropriate L2R and RL algorithms. Empirically, we evaluate our approach on extractive multi-document summarisation. We show that RELIS reduces the training time by two orders of magnitude compared to the state-of-the-art models while performing on par with them.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Fine-Tuning Language Models from Human Preferences (2019, methodology): “…such as BLEU for translation (Ranzato et al., 2015; Wu et al., 2016), ROUGE for summarization (Ranzato et al., 2015; Paulus et al., 2017; Wu and Hu, 2018; Gao et al., 2019b), music theory-based rewards (Jaques et al., 2017), or event detectors for story generation (Tambwekar et al., 2018).”
Learning to summarize from human feedback (2020, background): “A human judges which is a better summary of the post.”
Better Rewards Yield Better Summaries: Learning to Summarise Without References (2019, methodology): “Note that ExtAbsRL is a cross-input RL while NeuralTD is an input-speciﬁc RL (see § 2).”
RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (2023, unknown): “While initial applications of RL on tasks such as translation (Wu et al., 2016, 2018) and summarization (Gao et al., 2019; Wu and Hu, 2018) used automatic evaluation metrics as rewards, such simplified formulations of rewards did not fully align with human notions of quality.”
RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (2023, methodology): “While initial applications of RL on tasks such as translation (Wu et al., 2016; 2018) and summarization (Gao et al., 2019; Wu & Hu, 2018) used automatic evaluation metrics as rewards, such simpliﬁed formulations of rewards did not fully align with human notions of quality.”
Improving Factual Consistency Between a Response and Persona Facts (2021, background): “Optimizing for targeted quality factors is a principled approach to overcome these problems (e.g., Gao et al. (2019) optimize text summarization systems for quality factors relevant to that task).”
Survey on reinforcement learning for language processing (2021, methodology): “[38] approached another language processing task using reinforcement learning, namely document summarization.”
Preference-based interactive multi-document summarisation (2019, background): “Theoretically, our recent work (Gao et al. 2019) proves that RL-based text generators using learned reward are guaranteed to generate near-optimal outputs when both the reward learning module and the RL module are error-bounded.”