Strategy and Policy Learning for Non-Task-Oriented Conversational Systems

By Zhou Yu, Ziyu Xu, Alan W. Black, Alexander I. Rudnicky. 2016

102 citations recorded by the source corpus.

We propose a set of generic conversational strategies to handle possible sys-tem breakdowns in non-task-oriented dialog systems. We also design dialog policies to select among these strategies with respect to different dialog contexts. We combine expert knowledge and the statistical ﬁndings that derived from previous collected data in designing these policies. The dialog policy learned via reinforcement learning outperforms the random selection policy and the locally greedy policy in both the simulated and the real-world settings. In addition, we propose three metrics, which consider both the lo-cal and global quality of the conversation, to evaluate conversation quality.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluators (2019, methodology): “Li et al. (2016c) use features for information ﬂow, Yu et al. (2016) use features for turn-level appropriateness.”
Adversarial Learning for Neural Dialogue Generation (2017, methodology): “A similar strategy is adopted in Yu et al. (2016a).”
Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses (2017, background): “For example, (Li et al., 2016b) use features related to ease of answering and information flow, and (Yu et al., 2016) use metrics related to turn-level appropriateness and conversational depth.”
A Deep Reinforcement Learning Chatbot (2017, neutral): “The action-value function is also related to the the work by Yu et al. (2016), who learn an evaluation model, which is used to train a reinforcement learning agent to select appropriate dialogue response strategies.”
Generative Deep Neural Networks for Dialogue: A Short Review (2016, methodology): “Similarly, Yu et al. [2016] propose a hybrid model—combining retrieval models, neural networks and hand-crafted rules—trained using reinforcement learning to optimize a hand-crafted reward function.”
Challenges in Building Intelligent Open-domain Dialog Systems (2019, background): “To improve interactiveness, it is important to understand the user’s emotion state or affect [189, 192], to respond not only reactively but also proactively [108, 148, 169], to control the topic maintenance or transition [146], and to optimize the interaction strategy (i.”
A Repository of Conversational Datasets (2019, background): “This measures how often the correct response is identiﬁed as one of the top k ranked responses (Lowe et al., 2015; Inaba and Takahashi, 2016; Yu et al., 2016; Al-Rfou et al., 2016; Henderson et al., 2017; Lowe et al., 2017; Wu et al., 2017; Cer et al., 2018; Chaudhuri et al., 2018; Du and Black,…”
The Bottleneck Simulator: A Model-based Deep Reinforcement Learning Approach (2018, background): “As a more recent example, Yu et al. (2016) propose to learn a dialogue manager policy through model-free off-policy RL based on simulations with a rule-based system.”
A review of evaluation techniques for social dialogue systems (2017, methodology): “System-level rewards: The reward function by [3] was based on heuristics, whereas [13] use a Wizard-of-Oz experiment to measure engagement and deduct a reward function with the following metrics:”
Lexical Acquisition through Implicit Confirmations over Multiple Dialogues (2017, background): “Much attention has recently been paid to non-task-oriented dialogue systems —or chat-oriented dialogue systems— both in research (Higashinaka et al., 2014; Yu et al., 2016) and in industry.”
Persuasion for Good: Towards a Personalized Persuasive Dialogue System for Social Good (2019, background): “The primary goal of these agents is to facilitate taskcompletion and human-engagement in practical contexts (Luger and Sellen, 2016; Bickmore et al., 2016; Graesser et al., 2014; Yu et al., 2016b).”
Alana: Social Dialogue using an Ensemble Model and a Ranker trained on User Feedback (2017, methodology): “(Yu et al., 2016), we apply a ranker model to select the most relevant reply from a pool of replies generated by an ensemble of different agents/bots.”
Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders (2017, background): “Advanced dialog managers usually have a list of potential actions that enable them to have diverse behavior during a conversation, e.g. different strategies to recover from non-understanding (Yu et al., 2016).”