Better Rewards Yield Better Summaries: Learning to Summarise Without References

By Florian Böhm, Yang Gao, Christian M. Meyer, Ori Shapira, Ido Dagan, Iryna Gurevych. 2019

117 citations recorded by the source corpus.

Reinforcement Learning (RL)based document summarisation systems yield state-of-the-art performance in terms of ROUGE scores, because they directly use ROUGE as the rewards during training. However, summaries with high ROUGE scores often receive low human judgement. To find a better reward function that can guide RL to generate human-appealing summaries, we learn a reward function from human ratings on 2,500 summaries. Our reward function only takes the document and system summary as input. Hence, once trained, it can be used to train RL based summarisation systems without using any reference summaries. We show that our learned rewards have significantly higher correlation with human ratings than previous approaches. Human evaluation experiments show that, compared to the state-of-the-art supervised-learning systems and ROUGE-as-rewards RL summarisation systems, the RL systems using our learned rewards during training generate summaries with higher human ratings. The learned reward function and our source code are available at https://github.com/yg211/summary-reward-no-reference.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Training language models to follow instructions with human feedback (2022, methodology): “Originally developed for training simple robots in simulated environments and Atari games (Christiano et al., 2017; Ibarz et al., 2018), it has recently been applied to fine-tuning language models to summarize text (Ziegler et al., 2019; Stiennon et al., 2020; Böhm et al., 2019; Wu et al., 2021).”
Fine-Tuning Language Models from Human Preferences (2019, background): “In concurrent work, Böhm et al. (2019) also use human evaluations to learn a reward function for summarization, and optimize that reward function with RL.”
Learning to summarize from human feedback (2020, methodology): “[3] learn a reward function from a dataset of human ratings of 2.”
Recursively Summarizing Books with Human Feedback (2021, methodology): “This has been applied in many domains including summarization (Böhm et al., 2019; Ziegler et al., 2019; Stiennon et al., 2020), dialogue (Jaques et al., 2019; Yi et al., 2019; Hancock et al., 2019), translation (Kreutzer et al., 2018; Bahdanau et al., 2016), semantic parsing (Lawrence and Riezler,…”
MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance (2019, background): “…need for costly human references in the evaluation of text generation systems, and instead base evaluation scores on source texts and system predictions only, which would allow for ‘next-level’, unsupervised (in a double sense) and unlimited evaluation (Louis and Nenkova, 2013; B¨ohm et al., 2019).”
SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarization (2020, methodology): “Böhm et al. (2019) train a BERT-based evaluation function with 2,500 human ratings for 500 machinegenerated summaries from the CNN/DailyMail dataset; their method correlates better with human ratings than ROUGE and BLEU.”
Learning by Semantic Similarity Makes Abstractive Summarization Better (2020, methodology): “, 2018) or neural network (Böhm et al., 2019) as reward, showed remarkable performance.”
Automatic Text Evaluation through the Lens of Wasserstein Barycenters (2021, background): “, 2021) and many of these metrics can be further optimized to produce better summaries (Böhm et al., 2019).”
RRHF: Rank Responses to Align Language Models with Human Feedback without tears (2023, background): “Reinforcement learning techniques have been explored to align language models with human preferences [2, 4, 28, 13, 34, 21, 24].”
Offline RL for Natural Language Generation with Implicit Language Q Learning (2022, background): “A number of prior works have explored combining online RL methods with language models for natural language tasks such as machine translation or summarization [19, 20, 21, 22, 23, 24, 25, 26, 27, 28].”
Re-evaluating Evaluation in Text Summarization (2020, background): “For example, top-scoring reinforcement learning based summarization systems (Böhm et al., 2019) and the current state-of-the-art extractive system (Zhong et al.”
ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation (2023, methodology): “Previous work [5, 49] used human feedback to train reward functions for summarization tasks, while InstructGPT [36] applied RLHF to GPT-3 for multi-task NLP, yielding significant improvements.”
SummEval: Re-evaluating Summarization Evaluation (2020, unknown): “M18 - NeuralTD (Böhm et al., 2019) learn a reward function from 2,500 human judgments which is used in a reinforcement learning setting.”