Human-centric Dialog Training via Offline Reinforcement Learning

By Natasha Jaques, Judy Hanwen Shen, Asma Ghandeharioun, Craig Ferguson, Àgata Lapedriza, Noah J. Jones, Shixiang Shane Gu. 2020

120 citations recorded by the source corpus.

How can we train a dialog model to produce better conversations by learning from human feedback, without the risk of humans teaching it harmful chat behaviors? We start by hosting models online, and gather human feedback from real-time, open-ended conversations, which we then use to train and improve the models using offline reinforcement learning (RL). We identify implicit conversational cues including language similarity, elicitation of laughter, sentiment, and more, which indicate positive human feedback, and embed these in multiple reward functions. A well-known challenge is that learning an RL policy in an offline setting usually fails due to the lack of ability to explore and the tendency to make over-optimistic estimates of future reward. These problems become even harder when using RL for language models, which can easily have a 20,000 action vocabulary and many possible reward functions. We solve the challenge by developing a novel class of offline RL algorithms. These algorithms use KL-control to penalize divergence from a pre-trained prior language model, and use a new strategy to make the algorithm pessimistic, instead of optimistic, in the face of uncertainty. We test the resulting dialog model with ratings from 80 users in an open-domain setting and find it achieves significant improvements over existing deep offline RL approaches. The novel offline RL method is viable for improving any existing generative dialog model using a static dataset of human feedback.

Subfields: Artificial Intelligence, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Direct Preference Optimization: Your Language Model is Secretly a Reward Model (2023, methodology): “In particular, we formulate the following optimization problem max where � is a parameter controlling the deviation from the base reference policy ⇡ ref , namely the initial SFT model ⇡ SFT .”
Pretraining Language Models with Human Preferences (2023, methodology): “However, offline RL techniques were recently used for finetuning pretrained LMs to be aligned with human preferences in dialog tasks (Jaques et al., 2020; Jang et al., 2022; Snell et al., 2022).”
Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization (2022, background): “…et al., 2017; Kiegeland & Kreutzer, 2021), summarization (Stiennon et al., 2020; Paulus et al., 2017), dialogue (Li et al., 2016; Zhou et al., 2017; Jaques et al., 2020), image captioning (Rennie et al., 2017), question generation (Pang & He, 2021), text-games (Narasimhan et al., 2015; Hausknecht…”
Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs (2022, background): “emotional and affective states (Litman and ForbesRiley, 2004; Jaques et al., 2020), and incorporating empathy, interlocutor perspective, and social intelligence (Kearns et al.”
Can Wikipedia Help Offline Reinforcement Learning? (2022, background): “Numerous algorithmic work ensued (Wu et al., 2019; Jaques et al., 2020; Ghasemipour et al., 2021; Kumar et al., 2020; Fujimoto & Gu, 2021) with various applications (Jaques et al., 2020; Chebotar et al., 2021).”
Learning Natural Language Generation with Truncated Reinforcement Learning (2022, background): “Moreover, task-specific language models learned with SL suffer from well-studied text degeneration issues (Holtzman et al., 2019), such as the exposure bias (Bengio et al., 2015), language biases (Saleh et al., 2020; Jaques et al., 2020), or a lack of diversity (Li et al., 2015).”
Recent advances in deep learning based dialogue systems: a systematic survey (2021, unknown): “Majumder et al. (2020a) and Jaques et al. (2020) performed persona selection and persona-based response generation simultaneously and trained their agents with a reinforcement framework.”
Dynamic Planning in Open-Ended Dialogue using Reinforcement Learning (2022, background): “word level, so the action space is the entire vocabulary [14, 19, 20, 33].”
Offline RL for Natural Language Generation with Implicit Language Q Learning (2022, methodology): “[12, 13] remove this issue by defining actions at the “per-token” level, the offline RL algorithm proposed requires querying likelihoods from a language model at RL training time.”
Text Generation with Efficient (Soft) Q-Learning (2021, methodology): “Further, previous work has largely focused on the extreme of using only off-policy data, mostly for ofﬂine training of chatbots [23].”
Reinforcement Learning for Language Models (2023, methodology): “In the NLP context, RL has been used to improve model behavior in many tasks such as automatic translation [5, 6], question generation [7], summarization [8, 9], dialogue generation [10, 11], and text-based games [12, 13], to name a few.”