Guided Dialog Policy Learning: Reward Estimation for Multi-Domain Task-Oriented Dialog

By Ryuichi Takanobu, Hanlin Zhu, Minlie Huang. 2019

104 citations recorded by the source corpus.

Dialog policy decides what and how a task-oriented dialog system will respond, and plays a vital role in delivering effective conversations. Many studies apply Reinforcement Learning to learn a dialog policy with the reward function which requires elaborate design and pre-specified user goals. With the growing needs to handle complex goals across multiple domains, such manually designed reward functions are not affordable to deal with the complexity of real-world tasks. To this end, we propose Guided Dialog Policy Learning, a novel algorithm based on Adversarial Inverse Reinforcement Learning for joint reward estimation and policy optimization in multi-domain task-oriented dialog. The proposed approach estimates the reward signal and infers the user goal in the dialog sessions. The reward estimator evaluates the state-action pairs so that it can guide the dialog policy at each dialog turn. Extensive experiments on a multi-domain dialog dataset show that the dialog policy guided by the learned reward function achieves remarkably higher task success than state-of-the-art baselines.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Dynamic Dialogue Policy for Continual Reinforcement Learning (2022, methodology): “The approaches directly vectorise the belief state, hence the size of the input vector depends on the domain (as different domains for instance have different numbers of slots) (Su et al., 2016; Lipton et al., 2018; Weisz et al., 2018; Takanobu et al., 2019; Zhu et al., 2020).”
Semi-Supervised Dialogue Policy Learning via Stochastic Reward Estimation (2020, background): “Early studies focus on learning reward function utilizing external evaluations, e.g., user experience feedbacks (Gaˇsi´c et al., 2013), objective ratings (Su et al., 2015; Ultes et al., 2017), or a combination of multiple evaluations (Su et al., 2016; Chen et al., 2019).”
Dynamic Dialogue Policy Transformer for Continual Reinforcement Learning (2022, methodology): “Bin: The first baseline uses a flattened dialogue state for the state representation with binary values for every information which is the most common way (Takanobu et al., 2019; Zhu et al., 2020; Weisz et al., 2018).”
Offline Reinforcement Learning from Human Feedback in Real-World Sequence-to-Sequence Tasks (2020, background): “To this aim, inverse RL has been proposed to infer reward functions from responses indirectly (Takanobu et al., 2019).”
Domain-independent User Simulation with Transformers for Task-oriented Dialogue Systems (2021, result): “This was also observed in the experiments by Takanobu et al. (2019).”
ConvLab-2: An Open-Source Toolkit for Building, Evaluating, and Diagnosing Dialogue Systems (2020, background): “ConvLab-2 provides 1Please refer to the ConvLab-2 site for more details. a rule-based policy, a model-based policy where the model can be learned via supervised learning, or reinforcement learning methods such as REINFORCE (Williams, 1992), PPO (Schulman et al., 2017), and GDPL (Takanobu et al., 201”