Rethinking Action Spaces for Reinforcement Learning in End-to-end Dialog Agents with Latent Variable Models

By Tiancheng Zhao, Kaige Xie, Maxine Eskénazi. 2019

148 citations recorded by the source corpus.

Defining action spaces for conversational agents and optimizing their decision-making process with reinforcement learning is an enduring challenge. Common practice has been to use handcrafted dialog acts, or the output vocabulary, e.g. in neural encoder decoders, as the action spaces. Both have their own limitations. This paper proposes a novel latent action framework that treats the action spaces of an end-to-end dialog agent as latent variables and develops unsupervised methods in order to induce its own action space from the data. Comprehensive experiments are conducted examining both continuous and discrete action types and two different optimization methods based on stochastic variational inference. Results show that the proposed latent actions achieve superior empirical performance improvement over previous word-level policy gradient methods on both DealOrNoDeal and MultiWoz dialogs. Our detailed analysis also provides insights about various latent variable approaches for policy learning and can serve as a foundation for developing better latent actions in future research.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

A Simple Language Model for Task-Oriented Dialogue (2020, background): “[63] proposed to model system actions as latent variables, inducing a latent action space with variational inference methods.”
Hello, It’s GPT-2 - How Can I Help You? Towards the Use of Pretrained Language Models for Task-Oriented Dialogue Systems (2019, methodology): “Following prior work (Budzianowski et al., 2018b; Zhao et al., 2019; Chen et al., 2019), our evaluation task is the dialogue-context-to-text generation task (see Figure 1).”
Task-Oriented Dialog Systems that Consider Multiple Appropriate Responses under the Same Context (2019, background): “Although several models are proposed to handle the multi-domain response generation task (Zhao, Xie, and Eskenazi 2019; Mehri, Srinivasan, and Eskenazi 2019; Chen et al. 2019), the generation quality is far from perfect, presumably due to the complex task deﬁnitions, large policy coverage and…”
Structured Fusion Networks for Dialog (2019, methodology): “In dialog modelling, this problem is observed in the output of dialog models finetuned with reinforcement learning (Lewis et al., 2017; Zhao et al., 2019).”
SOLOIST: Few-shot Task-Oriented Dialog with A Single Pre-trained Auto-regressive Model (2020, methodology): “To tackle this, several models were proposed to handle the multi-domain dialog response generation (Pei et al., 2019; Budzianowski and Vuli´c, 2019; Mehri et al., 2019; Zhao et al., 2019; Wu et al., 2019; Zhang et al., 2019a).”
Survey on reinforcement learning for language processing (2021, background): “This approach can be used to train dialogue agents using reinforcement learning [138].”
Semi-Supervised Dialogue Policy Learning via Stochastic Reward Estimation (2020, background): “Dialogue policy is a critical component in both the conventional pipeline approach (Young et al., 2013) and recent end-to-end approaches (Zhao et al., 2019).”
ConvLab-2: An Open-Source Toolkit for Building, Evaluating, and Diagnosing Dialogue Systems (2020, methodology): “, 2017) and LaRL (Zhao et al., 2019) models.”
Dynamic Planning in Open-Ended Dialogue using Reinforcement Learning (2022, background): “Related approaches model actions as latent variables, inducing a latent action space, thus decoupling the discourse-level decision making fromNLG [30, 45].”
End-to-End Neural Pipeline for Goal-Oriented Dialogue Systems using GPT-2 (2020, background): “, 2019a) which maps the dialogue history to the dialogue state (the composite function of NLU and DST), and the Word-level POL (Budzianowski et al., 2018; Pei et al., 2019; Chen et al., 2019; Mehri et al., 2019; Zhao et al., 2019) which maps the previous utterance and dialogue state to the system re”
Improving Factual Consistency Between a Response and Persona Facts (2021, background): “As also suggested in prior work (Yarats and Lewis, 2018; Zhao et al., 2019; Bao et al., 2019), applying RL for speciﬁc metrics might bring in adverse impacts on linguistic quality.”
ConveRT: Efficient and Accurate Conversational Representations from Transformers (2019, background): “Data-driven task-oriented dialog systems require domain-speciﬁc labelled data: annotations for in-tents, explicit dialog states, and mentioned entities (Williams, 2014; Wen et al., 2017b,a; Ramadan et al., 2018; Liu et al., 2018; Zhao et al., 2019b).”
MultiWOZ 2. (2019, background): “…a unique resource, in terms of its multi-domain interactions as well as slot value transfers be-tween these domains, and has quickly attracted researchers for dialogue state tracking (Nouri and Hosseini-Asl, 2018; Goel et al., 2019; Wu et al., 2019) and dialogue policy learning (Zhao et al., 2019).”
MultiWOZ 2.1: Multi-Domain Dialogue State Corrections and State Tracking Baselines (2019, background): “…a unique resource, in terms of the multi-domain interactions as well as slot value transfers between these domains, and has quickly attracted researchers for dialogue state tracking (Nouri and Hosseini-Asl, 2018; Goel et al., 2019; Wu et al., 2019) and dialogue policy learning (Zhao et al., 2019).”
Alternating Recurrent Dialog Model with Large-scale Pre-trained Language Models (2019, methodology): “, 2019) model that incorporates dialog act supervision as an inductive prior for model architecture, and the LaRL (Zhao et al., 2019) model which leverages latent action modeling and reinforcement learning to improve performance.”