Hierarchical Reinforcement Learning for Open-Domain Dialog

By Abdelrhman Saleh, Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Rosalind W. Picard. 2019

61 citations recorded by the source corpus.

Open-domain dialog generation is a challenging problem; maximum likelihood training can lead to repetitive outputs, models have difficulty tracking long-term conversational goals, and training on standard movie or online datasets may lead to the generation of inappropriate, biased, or offensive text. Reinforcement Learning (RL) is a powerful framework that could potentially address these issues, for example by allowing a dialog model to optimize for reducing toxicity and repetitiveness. However, previous approaches which apply RL to open-domain dialog generation do so at the word level, making it difficult for the model to learn proper credit assignment for long-term conversational rewards. In this paper, we propose a novel approach to hierarchical reinforcement learning (HRL), VHRL, which uses policy gradients to tune the utterance-level embedding of a variational sequence model. This hierarchical approach provides greater flexibility for learning long-term, conversational rewards. We use self-play and RL to optimize for a set of human-centered conversation metrics, and show that our approach provides significant improvements – in terms of both human evaluation and automatic metrics – over state-of-the-art dialog models, including Transformers.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Red Teaming Language Models with Language Models (2022, background): “The target LM may also be trained using RL Saleh et al. (2020).”
Human-centric Dialog Training via Offline Reinforcement Learning (2020, methodology): “For model architectures, we focused on hierarchical sequence-to-sequence models (Serban et al., 2016, 2017b; Park et al., 2018) because they were found to be more effective for the datasets under consideration than e.g. Trans-formers (Saleh et al., 2019).”
Approximating Interactive Human Evaluation with Self-Play for Open-Domain Dialog Systems (2019, methodology): “If one intends to optimize M̂H , we suggest calculating it on conversation trajectories between the bot and an external baseline model or a fixed copy [41], or adopting adversarial learning by maintaining a discriminator to distinguish between real/fake conversations [42].”
Just Say No: Analyzing the Stance of Neural Dialogue Generation in Offensive Contexts (2021, background): “Saleh et al. (2020) proposes a toxicity classifier-based reinforcement learning objective to discourage the dialogue model from generating inappropriate responses.”
Learning Natural Language Generation with Truncated Reinforcement Learning (2022, background): “Moreover, task-specific language models learned with SL suffer from well-studied text degeneration issues (Holtzman et al., 2019), such as the exposure bias (Bengio et al., 2015), language biases (Saleh et al., 2020; Jaques et al., 2020), or a lack of diversity (Li et al., 2015).”
Dynamic Planning in Open-Ended Dialogue using Reinforcement Learning (2022, background): “Related approaches model actions as latent variables, inducing a latent action space, thus decoupling the discourse-level decision making fromNLG [30, 45].”