Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learning

By Julia Kreutzer, Joshua Uyheng, Stefan Riezler. 2018

95 citations recorded by the source corpus.

We present a study on reinforcement learning (RL) from human bandit feedback for sequence-to-sequence learning, exemplified by the task of bandit neural machine translation (NMT). We investigate the reliability of human bandit feedback, and analyze the influence of reliability on the learnability of a reward estimator, and the effect of the quality of reward estimates on the overall RL task. Our analysis of cardinal (5-point ratings) and ordinal (pairwise preferences) feedback shows that their intra- and inter-annotator α-agreement is comparable. Best reliability is obtained for standardized cardinal feedback, and cardinal feedback is also easiest to learn and generalize from. Finally, improvements of over 1 BLEU can be obtained by integrating a regression-based reward estimator trained on cardinal feedback for 800 translations into RL for NMT. This shows that RL is possible even from small amounts of fairly reliable human feedback, pointing to a great potential for applications at larger scale.

Subfields: Artificial Intelligence, Management Science and Operations Research, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Direct Preference Optimization: Your Language Model is Secretly a Reward Model (2023, background): “Despite the success of instruction tuning, relative human judgments of response quality are often easier to collect than expert demonstrations, and thus subsequent works have fine-tuned LLMs with datasets of human preferences, improving proficiency in translation [16], summarization [35, 45], story-”
Fine-Tuning Language Models from Human Preferences (2019, methodology): “Similarly, Kryściński et al. (2019) found that 30% of samples from the supervised summarization models they tested contained inconsistencies, and Khandelwal et al.”
Better Rewards Yield Better Summaries: Learning to Summarise Without References (2019, methodology): “(2)), our method can even learn the reward function from preferences over generated summaries, an even cheaper feedback to elicit (Kreutzer et al., 2018; Gao et al., 2018).”
Scalable agent alignment via reward modeling: a research direction (2018, background): “This includes trajectory preferences (Christiano et al., 2017; Kreutzer et al., 2018), goal state examples (Bahdanau et al., 2018), demonstrations (Finn et al., 2016; Ho & Ermon, 2016), as well as combinations thereof (Tung et al., 2018; Ibarz et al., 2018).”
Modeling Fluency and Faithfulness for Diverse Neural Machine Translation (2019, background): “This series of work (Wu et al. 2018; Yang et al. 2018; Geng et al. 2018; Kreutzer, Uyheng, and Riezler 2018) samples a translation from all the possible translation and performs gradient descent through the translation with a sequence-level reward under the framework of reinforcement learning .”
Statistical Rejection Sampling Improves Preference Optimization (2023, background): “This approach has led to enhanced performance in various applications, including translation (Kreutzer et al., 2018), summarization (Stiennon et al., 2020), storytelling (Ziegler et al., 2019), and following instructions (Ramamurthy et al., 2022).”
Offline Reinforcement Learning from Human Feedback in Real-World Sequence-to-Sequence Tasks (2020, methodology): “2 Furthermore, interpreting feedback wrongly (e.g., through incorrect credit assignment (Bah-danau et al., 2017)), or receiving misleading feed-back (Nguyen et al., 2017; Kreutzer et al., 2018a), could easily push the policy into less favorable conditions.”
Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive (2024, methodology): “These techniques have been used to improve performance on a variety of downstream tasks such as translation [Kreutzer et al., 2018] and summarisation [Stiennon et al., 2020], as well as to create general-purpose models such as Zephyr [Tunstall et al., 2023].”
Informed Machine Learning – A Taxonomy and Survey of Integrating Prior Knowledge into Learning Systems (2019, background): “ing approach is learning a reward estimator [151], [152] from collected logs, which then provides unlimited feedback for unseen instances that do not have any human judgments.”
Model Alignment as Prospect Theoretic Optimization (2024, methodology): “LLM Alignment Human feedback has been used to improve LLM capabilities in translation (Kreutzer et al., 2018), summarization (Stiennon et al., 2020), sentiment-conditioned generation (Ziegler et al., 2019), and instruction-following (Ouyang et al., 2022).”
Preference-based interactive multi-document summarisation (2019, methodology): “Interactive Natural Language Processing (NLP) approaches that put the human in the loop gained increasing research interests recently (Amershi et al. 2014; Gurevych et al. 2018; Kreutzer et al. 2018a).”