Feudal Reinforcement Learning for Dialogue Management in Large Domains

By Iñigo Casanueva, Paweł Budzianowski, Pei-hao Su, Stefan Ultes, Lina Maria Rojas-Barahona, Bo-Hsiang Tseng, Milica Gasic. 2018

50 citations recorded by the source corpus.

Reinforcement learning (RL) is a promising approach to solve dialogue policy optimisation. Traditional RL algorithms, however, fail to scale to large domains due to the curse of dimensionality. We propose a novel Dialogue Management architecture, based on Feudal RL, which decomposes the decision into two steps; a first step where a master policy selects a subset of primitive actions, and a second step where a primitive action is chosen from the selected subset. The structural information included in the domain ontology is used to abstract the dialogue state space, taking the decisions at each step using different parts of the abstracted state. This, combined with an information sharing mechanism between slots, increases the scalability to large domains. We show that an implementation of this approach, based on Deep-Q Networks, significantly outperforms previous state of the art in several dialogue domains and environments, without the need of any additional reward signal.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Dynamic Dialogue Policy for Continual Reinforcement Learning (2022, methodology): “In the realm of domain adaptation, works assume a fixed number of actions that are slot-independent, and focus on the inclusion of slot-dependent actions when the domain changes (Wang et al., 2015; Casanueva et al., 2018; Chen et al., 2018; Chen et al., 2020; Lin et al., 2021).”
Neural Approaches to Conversational AI (2018, unknown): “Finally, another interesting attempt is made by Casanueva et al. (2018) based on Feudal Reinforcement Learning (FRL) (Dayan and Hinton, 1993).”
Guided Dialog Policy Learning: Reward Estimation for Multi-Domain Task-Oriented Dialog (2019, background): “…addressed by hierarchical RL which decomposes the task into several sub-tasks in terms of temporal order (Peng et al., 2017) or space abstraction (Casanueva et al., 2018), but the hierarchical structure can be very complex and constraints between different domains should be considered if an…”
Dynamic Dialogue Policy Transformer for Continual Reinforcement Learning (2022, methodology): “In modular task-oriented dialogue systems the input to the dialogue policy can be modelled in many different ways (Lipton et al., 2018; Weisz et al., 2018; Takanobu et al., 2019; Wang et al., 2015; Casanueva et al., 2018; Xu et al., 2020).”