Sample Efficient Deep Reinforcement Learning for Dialogue Systems With Large Action Spaces

By Gellért Weisz, Paweł Budzianowski, Pei-hao Su, Milica Gasic. 2018

87 citations recorded by the source corpus.

In spoken dialogue systems, we aim to deploy artificial intelligence to build automated dialogue agents that can converse with humans. A part of this effort is the policy optimization task, which attempts to find a policy describing how to respond to humans, in the form of a function taking the current state of the dialogue and returning the response of the system. In this paper, we investigate deep reinforcement learning approaches to solve this problem. Particular attention is given to actor-critic methods, off-policy reinforcement learning with experience replay, and various methods aimed at reducing the bias and variance of estimators. When combined, these methods result in the previously proposed ACER algorithm that gave competitive results in gaming environments. These environments, however, are fully observable and have a relatively small action set so, in this paper, we examine the application of ACER to dialogue policy optimization. We show that this method beats the current state of the art in deep learning approaches for spoken dialogue systems. This not only leads to a more sample efficient algorithm that can train faster, but also allows us to apply the algorithm in more difficult environments than before. We thus experiment with learning in a very large action space, which has two orders of magnitude more actions than previously considered. We find that ACER trains significantly faster than the current state of the art.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Dynamic Dialogue Policy for Continual Reinforcement Learning (2022, methodology): “The approaches directly vectorise the belief state, hence the size of the input vector depends on the domain (as different domains for instance have different numbers of slots) (Su et al., 2016; Lipton et al., 2018; Weisz et al., 2018; Takanobu et al., 2019; Zhu et al., 2020).”
Dynamic Dialogue Policy Transformer for Continual Reinforcement Learning (2022, methodology): “Bin: The first baseline uses a flattened dialogue state for the state representation with binary values for every information which is the most common way (Takanobu et al., 2019; Zhu et al., 2020; Weisz et al., 2018).”
Uncertainty Measures in Neural Belief Tracking and the Effects on Dialogue Policy Performance (2021, methodology): “This is also possible using neural approaches with less computational effort (Jurˇcíˇcek et al., 2011; Weisz et al., 2018; Chen et al., 2020).”
An Overview of Natural Language State Representation for Reinforcement Learning (2020, background): “A common approach was building a belief state (a distribution over possible dialogue states, usually represented by a fixed set of slot-value pairs) when dialogue tasks are modeled as POMDPs (Su et al., 2016; Fatemi et al., 2016; Wen et al., 2017b; Weisz et al., 2018).”
Distributed Structured Actor-Critic Reinforcement Learning for Universal Dialogue Management (2020, methodology): “First, we compare the performance of STRAC-M with the performance of ACER, STRAC-S and DQNDIP-M.”