Composite Task-Completion Dialogue Policy Learning via Hierarchical Deep Reinforcement Learning

By Baolin Peng, Xiujun Li, Lihong Li, Jianfeng Gao, Asli Celikyilmaz, Sungjin Lee, Kam-Fai Wong. 2017

195 citations recorded by the source corpus.

Building a dialogue agent to fulfill complex tasks, such as travel planning, is challenging because the agent has to learn to collectively complete multiple subtasks. For example, the agent needs to reserve a hotel and book a flight so that there leaves enough time for commute between arrival and hotel check-in. This paper addresses this challenge by formulating the task in the mathematical framework of options over Markov Decision Processes (MDPs), and proposing a hierarchical deep reinforcement learning approach to learning a dialogue manager that operates at different temporal scales. The dialogue manager consists of: (1) a top-level dialogue policy that selects among subtasks or options, (2) a low-level dialogue policy that selects primitive actions to complete the subtask given by the top-level policy, and (3) a global state tracker that helps ensure all cross-subtask constraints be satisfied. Experiments on a travel planning task with simulated and real users show that our approach leads to significant improvements over three baselines, two based on handcrafted rules and the other based on flat deep reinforcement learning.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jianfeng Gao
2. Bing Liu
3. Erik Cambria
4. Minlie Huang
5. Ian Lane
6. Sergey Levine
7. Percy Liang
8. Li Deng

Citations in the corpus

• End-to-End Offline Goal-Oriented Dialog Policy Learning via Policy Gradient (2017, background): “Recently, many RL-based algorithms have been proposed and achieved good results [15, 5, 30, 22].”
• Dialogue Learning with Human Teaching and Feedback in End-to-End Trainable Task-Oriented Dialogue Systems (2018, background): “Both supervised learning (SL) based (Wen et al., 2017; Bordes and Weston, 2017; Liu and Lane, 2017a) and deep reinforcement learning (RL) based systems (Zhao and Es-kenazi, 2016; Li et al., 2017; Peng et al., 2017) have been studied in the literature.”
• Challenges in Building Intelligent Open-domain Dialog Systems (2019, background): “3Dialog management performs both dialog state tracking [47, 89] and response selection via policy [71, 99, 132, 183].”
• Hierarchical Imitation and Reinforcement Learning (2018, background): “For instance, a chatbot assisting a user with reservation and booking for flights and hotels (Peng et al., 2017; El Asri et al., 2017) needs to navigate through multiple turns of conversation.”
• Emotionally-Aware Chatbots: A Survey (2019, background): “Nowadays, chatbots are mostly used as customer service such as booking systems [38, 59], shopping assistance [10] or just as conversational partner such as Endurance 4 and Insomnobot 5.”
• A Teacher-Student Framework for Maintainable Dialog Manager (2018, methodology): “For example, an inform slot , such as “ inform(cuisine=Sichuan cuisine) ”, indicates the user finding a Sichuan restaurant, and a request slot , such as “ request(area) ”, indicates the user asking for information from the system (Li et al., 2016, 2017b; Peng et al., 2017).”
• Meta Dialogue Policy Learning (2020, methodology): “This is due to the inherent difficulties in cross-domain dialogue learning: i) the state-action space to explore is much larger, and ii) the conversation required to complete the task is often longer [2].”
• Structured Dialogue Policy with Graph Neural Networks (2018, methodology): “…methods, like deep Q-network (DQN) (Cuay´ahuitl et al., 2015; Zhao and Eskenazi, 2016; Fatemi et al., 2016; Chen et al., 2017a; Chang et al., 2017; Peng et al., 2017), and policy gradient methods, e.g. REINFORCE (Williams et al., 2017), advantage actor-critic (A2C) (Fatemi et al., 2016).”
• Multi-Level Memory for Task Oriented Dialogs (2018, unknown): “, 2017) and dialogue act prediction (Peng et al., 2017; Tang et al., 2018).”
• Guided Dialog Policy Learning: Reward Estimation for Multi-Domain Task-Oriented Dialog (2019, methodology): “Among many models, Reinforcement Learning (RL) is commonly used to learn dialog policy (Fatemi et al., 2016; Peng et al., 2017; Chen et al., 2017; Yarats and Lewis, 2018; Lei et al., 2018; He et al., 2018; Su et al., 2018), where users are modeled as a part of the environment and the policy is…”
• Distributed Structured Actor-Critic Reinforcement Learning for Universal Dialogue Management (2020, methodology): “It is usually formulated by options framework [15] and solved using hierarchical reinforcement learning methods [14], [16], [17].”
• Neural Approaches to Conversational AI (2018, unknown): “Many published studies involving human subjects are of the first type (Walker, 2000; Singh et al., 2002; Ai et al., 2007; Rieser and Lemon, 2011; Gašić et al., 2013; Wen et al., 2015; Young et al., 2016; Peng et al., 2017; Lipton et al., 2018).”
• Hierarchical Reinforcement Learning for Open-Domain Dialog (2019, background): “However, attempts to apply hierarchical RL (HRL) to dialog generation have so far been limited to task-oriented dialog systems (Zhang, Zhao, and Yu 2018; Peng et al. 2017; Budzianowski et al. 2017; Tang et al. 2018).”
• Adversarial Learning of Task-Oriented Neural Dialog Models (2018, methodology): “Instead of using a rule-based user simulator as in many prior work (Zhao and Eskenazi, 2016; Peng et al., 2017), in our study we use a model-based simulator trained on DSTC2 dataset.”