APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning

By Yang Gao, Christian M. Meyer, Iryna Gurevych. 2018

34 citations recorded by the source corpus.

We propose a method to perform automatic document summarisation without using reference summaries. Instead, our method interactively learns from users’ preferences. The merit of preference-based interactive summarisation is that preferences are easier for users to provide than reference summaries. Existing preference-based interactive learning methods suffer from high sample complexity, i.e. they need to interact with the oracle for many rounds in order to converge. In this work, we propose a new objective function, which enables us to leverage active learning, preference learning and reinforcement learning techniques in order to reduce the sample complexity. Both simulation and real-user experiments suggest that our method significantly advances the state of the art. Our source code is freely available at https://github.com/UKPLab/emnlp2018-april.

Subfields: Artificial Intelligence, Natural Language Processing, Reinforcement Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Reinforcement Learning

In Signal Processing

Citations in the corpus

Better Rewards Yield Better Summaries: Learning to Summarise Without References (2019, methodology): “ilar idea has been explored by Gao et al. (2019), but unlike their work that learns the reward from ROUGE scores, we learn our reward directly from human ratings.”
Quark: Controllable Text Generation with Reinforced Unlearning (2022, background): “Recent works have explored applying RL on tasks such as question-answering [84, 85, 47, 83, 84], summarization [58, 53, 70, 60, 17, 51], and machine translation [58, 87, 79, 82, 81, 13, 66, 5, 49].”
Best Practices for Crowd-based Evaluation of German Summarization: Comparing Crowd, Expert and Automatic Evaluation (2020, background): “Following, Gao et al. (2018); Falke et al. (2017); Fan et al. (2018) have used crowdsourcing as the source of human evaluation to rate their automatic summarization systems.”
Preference-based interactive multi-document summarisation (2019, methodology): “(ii) We evaluate multiple new APL strategies and a novel neural RL algorithm, and compare them with the counterpart methods used in Gao et al. (2018).”
Reward Learning for Efficient Reinforcement Learning in Extractive Document Summarisation (2019, background): “The R2 scores for optimal extractive summaries are usually half of their R1 scores [Gao et al., 2018], so we multiply R2 by 2 to balance the impact of R1 and R2.”
SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarization (2020, background): “Besides SCUs, recent work has explored eliciting preferences over summaries (Zopf, 2018; Gao et al., 2018, 2019a) and annotations of important bi-grams (P.V.S and Meyer, 2017) to derive summary ratings.”
Crowdsourcing Lightweight Pyramids for Manual Summary Evaluation (2019, methodology): “Another common crowdsourcable evaluation method is pairwise comparison (Gao et al., 2018; Falke et al., 2017; Fan et al., 2018): an evaluator is asked to judge which of two competing summaries of the same text is superior, usually while”
Informed Machine Learning – A Taxonomy and Survey of Integrating Prior Knowledge into Learning Systems (2019, background): “ing approach is learning a reward estimator [151], [152] from collected logs, which then provides unlimited feedback for unseen instances that do not have any human judgments.”
Constrained Abstractive Summarization: Preserving Factual Consistency with Constrained Generation (2020, background): “Second, for human-in-the-loop interactive summarization (Gao et al., 2018; Shapira et al., 2020), when a model generates a summary that contains factual”