Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog

By Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Àgata Lapedriza, Noah J. Jones, Shixiang Shane Gu, Rosalind W. Picard. 2019

397 citations recorded by the source corpus.

Most deep reinforcement learning (RL) systems are not able to learn effectively from off-policy data, especially if they cannot explore online in the environment. These are critical shortcomings for applying RL to real-world problems where collecting data is expensive, and models must be tested offline before being deployed to interact with the environment -- e.g. systems that learn from human interaction. Thus, we develop a novel class of off-policy batch RL algorithms, which are able to effectively learn offline, without exploring, from a fixed batch of human interaction data. We leverage models pre-trained on data as a strong prior, and use KL-control to penalize divergence from this prior during RL training. We also use dropout-based uncertainty estimates to lower bound the target Q-values as a more efficient alternative to Double Q-Learning. The algorithms are tested on the problem of open-domain dialog generation -- a challenging reinforcement learning problem with a 20,000-dimensional action space. Using our Way Off-Policy algorithm, we can extract multiple different reward functions post-hoc from collected human interaction data, and learn effectively from all of these. We test the real-world generalization of these systems by deploying them live to converse with humans in an open-domain setting, and demonstrate that our algorithm achieves significant improvements over prior methods in off-policy batch RL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Training language models to follow instructions with human feedback (2022, methodology): “This work is in turn influenced by similar work using human feedback as a reward in domains such as dialogue (Jaques et al., 2019; Yi et al., 2019; Hancock et al., 2019), translation (Kreutzer et al., 2018; Bahdanau et al., 2016), semantic parsing (Lawrence and Riezler, 2018), story generation…”
Fine-Tuning Language Models from Human Preferences (2019, methodology): “Experiments using human evaluations as rewards include Kreutzer et al. (2018) which used off-policy reward learning for translation, and Jaques et al. (2019) which applied the modiﬁed Q-learning methods of Jaques et al. (2017) to implicit human preferences in dialog.”
Learning to summarize from human feedback (2020, methodology): “Human feedback has also been used as a reward to train models in other domains such as dialogue [25, 68, 21], translation [32, 1], semantic parsing [34], story generation [72], review generation [7], and evidence extraction [46].”
Recursively Summarizing Books with Human Feedback (2021, methodology): “This has been applied in many domains including summarization (Böhm et al., 2019; Ziegler et al., 2019; Stiennon et al., 2020), dialogue (Jaques et al., 2019; Yi et al., 2019; Hancock et al., 2019), translation (Kreutzer et al., 2018; Bahdanau et al., 2016), semantic parsing (Lawrence and Riezler,…”
Red Teaming Language Models with Language Models (2022, methodology): “To prevent RL from collapsing to a single, highreward generation, we add a loss term to penalize KL divergence between pr(x) and initialization’s distribution over next tokens (Jaques et al., 2017; Schmitt et al., 2018; Jaques et al., 2019; Ziegler et al., 2019).”
Aligning Language Models with Preferences through f-divergence Minimization (2023, methodology): “Furthermore, even if RL with KL penalties (Todorov, 2006a; Kappen et al., 2012; Jaques et al., 2017; 2019), the method used to fine-tune a LM in RLHF, is defined only in terms of reward maximization, it has also been shown to be equivalent to minimizing the reverse KL divergence KL(πθ||p) of πθ to…”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “An equivalent formulation adds the penalty term αD(π(·|s), πβ(·|s)) directly to the target Q-values and the actor objective (Wu et al., 2019a; Jaques et al., 2019), resulting in the following algorithm:”
Conservative Q-Learning for Offline Reinforcement Learning (2020, methodology): “Prior works have attempted to solve this problem by constraining the learned policy to be “close” to the behavior policy, for example as measured by KL-divergence [27, 59, 48, 54], Wasserstein distance [59], or MMD [30], and then only using actions sampled from this constrained policy in the Bellman”
Accelerating Online Reinforcement Learning with Offline Datasets (2020, neutral): “e sampling [41, 47, 26, 5, 17, 12]. Another class of methods perform ofﬂine reinforcement learning via dynamic programming, with an explicit constraint to prevent deviation from the data distribution [21, 8, 22, 46, 16]. While these algorithms perform well in the purely ofﬂine settings, we show in S”
MOPO: Model-based Offline Policy Optimization (2020, methodology): “To do so, a wide range of ofﬂine RL methods propose to mitigate bootstrapped error by constraining the learned policy to the behavior policy induced by the dataset [18, 35, 70, 29, 48, 51, 57].”
MOReL : Model-Based Offline Reinforcement Learning (2020, methodology): “[12], and has applications in healthcare [33, 34, 35], recommendation systems [36, 37, 38, 39], dialogue systems [40, 19, 41], and autonomous driving [42].”
D4RL: Datasets for Deep Data-Driven Reinforcement Learning (2020, methodology): “ntrollers. Ofﬂine reinforcement learning using large datasets has also been used in real-world systems where evaluation using a simulator is not possible, such as in robotics [3] and dialogue systems [15, 30, 16]. Moreover, signiﬁcant efforts have been made to incorporate large-scale datasets into o”
Behavior Regularized Offline Reinforcement Learning (2019, methodology): “As another example, when regularizing towards the behavior policy, Jaques et al. (2019) uses Kullback-Leibler (KL) divergence with a ﬁxed regularization weight while Kumar et al. (2019) proposes to use Maximum Mean Discrepancy (MMD) with an adaptively trained regularization weight.”
Striving for Simplicity in Off-policy Deep Reinforcement Learning (2019, background): “While there has been increasing interest in batch off-policy RL over the last few years [28, 56, 17, 25, 27], much of this focussed on off-policy policy evaluation, where the goal is to estimate the performance of a given target policy.”
Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction (2019, methodology): “…a baseline actor-critic algorithm (TD3), the BCQ algorithm [13], which aims to address a similar problem, as discussed in Section 4, KL-control [22] (which solves a KL-penalized RL problem similarly to maximum entropy RL), a static version of DQfD [21] (where a constraint to upweight Q-values…”
Offline RL Without Off-Policy Evaluation (2021, background): “Another proposal is to regularize the learned policy towards the behavior policy [Wu et al., 2019] usually either with a KL divergence [Jaques et al., 2019] or MMD [Kumar et al., 2019].”
A Minimalist Approach to Offline Reinforcement Learning (2021, methodology): “The solution class for this problem revolves around the idea that the learned policy should be kept close to the data-generating process (or behavior policy), and has been given a variety of names (such as batchconstrained [Fujimoto et al., 2019b], KL-control [Jaques et al., 2019], behavior-regulari”