Sample-efficient Actor-Critic Reinforcement Learning with Supervised Data for Dialogue Management

By Pei-hao Su, Paweł Budzianowski, Stefan Ultes, Milica Gasic, Susannah Young. 2017

135 citations recorded by the source corpus.

Deep reinforcement learning (RL) methods have significant potential for dialogue policy optimisation. However, they suffer from a poor performance in the early stages of learning. This is especially problematic for on-line learning with real users. Two approaches are introduced to tackle this problem. Firstly, to speed up the learning process, two sample-efficient neural networks algorithms: trust region actor-critic with experience replay (TRACER) and episodic natural actor-critic with experience replay (eNACER) are presented. For TRACER, the trust region helps to control the learning step size and avoid catastrophic model changes. For eNACER, the natural gradient identifies the steepest ascent direction in policy space to speed up the convergence. Both models employ off-policy learning with experience replay to improve sample-efficiency. Secondly, to mitigate the cold start issue, a corpus of demonstration data is utilised to pre-train the models prior to on-line reinforcement learning. Combining these two approaches, we demonstrate a practical approach to learn deep RL-based dialogue policies and demonstrate their effectiveness in a task-oriented information seeking domain.

Subfields: Artificial Intelligence, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Human-centric Dialog Training via Offline Reinforcement Learning (2020, background): “Improving dialog systems with RL has largely been restricted to task-oriented dialog systems, which have a limited number of task-specific actions (Fatemi et al., 2016; Gašić et al., 2011; Liu and Lane, 2017; Liu et al., 2018; Su et al., 2017).”
Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learning (2017, methodology): “We use advantage actor-critic (A2C) for policy optimization with off-policy learning (Su et al., 2017).”
End-to-End Offline Goal-Oriented Dialog Policy Learning via Policy Gradient (2017, methodology): “Usually, these algorithms are followed by training with reinforcement learning to further refine the model [21, 28, 23, 27].”
Dialogue Learning with Human Teaching and Feedback in End-to-End Trainable Task-Oriented Dialogue Systems (2018, background): “…components for natural language understanding (NLU) (Mesnil et al., 2015; Liu and Lane, 2016; Hakkani-T¨ur et al., 2016), dialogue state tracking (DST) (Henderson et al., 2014c; Mrkˇsi´c et al., 2016), and dialogue policy learning (Gasic and Young, 2014; Shah et al., 2016; Su et al., 2016, 2017).”
Structured Dialogue Policy with Graph Neural Networks (2018, methodology): “So the focus of these recent advances has been on designing improved RL algorithms to improve the sample-efficiency, e.g. eNAC (Su et al., 2017), BBQ (Lipton et al., 2018).”
Collaborative Multi-Agent Dialogue Model Training Via Reinforcement Learning (2019, methodology): “As a reward signal for reinforcement learning we use the standard reward function found in the literature (Gasic et al., 2013; Su et al., 2017, e.g.), tweaked to fit each agent’s perception as described in section 2.2.”
Feudal Reinforcement Learning for Dialogue Management in Large Domains (2018, unknown): “(Su et al., 2017).”
A Teacher-Student Framework for Maintainable Dialog Manager (2018, methodology): “Though there are many methods to train a RL-based dialog manager ef-ﬁciently (Su et al., 2016a, 2017; Lipton et al., 2017; Chang et al., 2017; Chen et al., 2017), the unmaintainable RL-based dialog systems will still be put on the shelf in real-world applications (Paek and Pieraccini, 2008; Paek,…”
Adversarial Learning of Task-Oriented Neural Dialog Models (2018, background): “Further more, online dialog policy learning with RL usually suffers from sample efﬁciency issue (Su et al., 2017), which requires an agent to make a large number of feedback queries to users.”
Sample Efficient Deep Reinforcement Learning for Dialogue Systems With Large Action Spaces (2018, methodology): “This implementation of ACER is also able to train efficiently in the master action space, showing the best performance among NN-based policy optimisers, as reported by [6] and [5].”
The Rapidly Changing Landscape of Conversational Agents (2018, background): “In the paper by Su et al, 2017 [28], they proposed a sample-efficient actor-critic reinforcement learning with supervised data for dialogue management.”
Hierarchical Reinforcement Learning for Open-Domain Dialog (2019, methodology): “Improving dialog models with RL is a difﬁcult problem, and past work has largely been restricted to task-oriented dialog systems, which have a limited number of task-speciﬁc actions (e.g. Liu and Lane; Su et al. (2017; 2017)).”
Rethinking Action Spaces for Reinforcement Learning in End-to-end Dialog Agents with Latent Variable Models (2019, methodology): “RL techniques are then used to find the optimal dialog policy (Gasic and Young, 2014; Su et al., 2017; Williams et al., 2017).”