Spoken Dialogue Management Using Probabilistic Reasoning

By Nicholas A. Roy, Joelle Pineau, Sebastian Thrun. 2000

337 citations recorded by the source corpus.

Spoken dialogue managers have benefited from using stochastic planners such as Markov Decision Processes (MDPs). However, so far, MDPs do not handle well noisy and ambiguous speech utterances. We use a Partially Observable Markov Decision Process (POMDP)-style approach to generate dialogue strategies by inverting the notion of dialogue state; the state represents the user's intentions, rather than the system state. We demonstrate that under the same noisy conditions, a POMDP dialogue manager makes fewer mistakes than an MDP dialogue manager. Furthermore, as the quality of speech recognition degrades, the POMDP dialogue manager automatically adjusts the policy.

Subfields: Artificial Intelligence, Social Psychology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Social Psychology

Citations in the corpus

Dynamic Dialogue Policy for Continual Reinforcement Learning (2022, background): “…dialogue systems have a tracking component that maintains information about the dialogue in a belief state, and a planning component that models the underlying policy, i.e., the selection of actions (Levin and Pieraccini, 1997; Roy et al., 2000; Williams and Young, 2007; Zhang et al., 2020b).”
Continuously Learning Neural Dialogue Management (2016, background): “More recently, this dialogue management task has been formulated as a reinforcement learning (RL) problem which can be automatically optimised through human interaction (Levin and Pierac-cini, 1997; Roy et al., 2000; Williams and Young, 2007; Jurˇc´ıˇcek et al., 2011; Young et al., 2013).”
Reinforcement Learning for Mapping Instructions to Actions (2009, methodology): “Reinforcement Learning for Language Processing Reinforcement learning has been previously applied to the problem of dialogue management (Scheffler and Young, 2002; Roy et al., 2000; Litman et al., 2000; Singh et al., 1999).”
Hybrid Reinforcement/Supervised Learning of Dialogue Policies from Fixed Data Sets (2008, background): “…representations and limited sets of actions to choose among (Walker, Fromer, and Narayanan 1998; Goddeau and Pineau 2000; Levin, Pieraccini, and Eckert 2000; Roy, Pineau, and Thrun 2000; Schefﬂer and Young 2002; Singh et al. 2002; Williams and Young 2005; Williams, Poupart, and Young 2005a).”
Sample-efficient Actor-Critic Reinforcement Learning with Supervised Data for Dialogue Management (2017, neutral): “More recently, it has been formulated as a planning problem and solved using reinforcement learning (RL) to optimise a dialogue policy through interaction with users (Levin and Pierac-cini, 1997; Roy et al., 2000; Williams and Young, 2007; Jurˇc´ıˇcek et al., 2011).”
The Hidden Information State model: A practical framework for POMDP-based spoken dialogue management (2010, background): “6 Note that alternative POMDP formulations can also be used for SDS (e.g. Roy et al., 2000; Zhang et al., 2001).”
Automating spoken dialogue management design using machine learning: An industry perspective (2008, background): “Furthermore, keeping track of unobservable states such as the user’s intentions and beliefs, which can be inferred from observations, such as the user’s utterance, has been shown to improve performance (e.g., Roy et al., 2000; Williams and Young, 2005; Zhang et al., 2001).”
Sample Efficient Deep Reinforcement Learning for Dialogue Systems With Large Action Spaces (2018, methodology): “To overcome this, the policy optimisation problem has been formulated as a reinforcement learning problem [1]–[3].”
Reinforcement Learning for Spoken Dialogue Systems: Comparing Strengths and Weaknesses for Practical Deployment (2006, background): “Furthermore, keeping track of unobservable states such as the user’s intentions and beliefs, which can be inferred from observations such as the user’s utterance, has been shown to improve performance [6, 7, 8].”
Reinforcement Learning with Limited Reinforcement : Using Bayes Risk for Active Learning in POMDPs Finale (2012, background): “To date, POMDP-based planners have been us d for applications as diverse as robot navigation [1, 2], ecological monitorin g [3], dynamic pricing [4], and several areas of dialog management and healthca re [5, 6, 7].”
Machine Learning for Dialog State Tracking: A Review (2015, methodology): “Early generative approaches used exact inference, enumerating all possible dialog states [14, 15].”
POMDP-Based Statistical Spoken Dialog Systems: A Review (2013, background): “framework of partially observable Markov decision processes (POMDPs(2)) [9]–[11].”
Dynamic Dialogue Policy Transformer for Continual Reinforcement Learning (2022, background): “…dialogue systems have a tracking component that maintains information about the dialogue in a belief state, and a planning component that models the underlying policy, i.e., the selection of actions (Levin and Pieraccini, 1997; Roy et al., 2000; Williams and Young, 2007; Zhang et al., 2020b).”
POMDP-based dialogue manager adaptation to extended domains (2013, background): “This provides increased robustness to errors in speech understanding and automatic dialogue policy optimisation via reinforcement learning (Roy et al., 2000; Zhang et al., 2001; Williams and Young, 2007; Young et al., 2010; Thomson and Young, 2010).”
Development and evaluation of interactive humanoid robots (2004, background): “Basically, the transition among the situated modules is implemented as follows: the robot sometimes asks humans for interaction by saying, “Let’s play, touch me,” and exhibits idling and moving-around behaviors until a human acts in response; once a human reacts to the robot (touches or speaks), it…”
Reinforcement learning with limited reinforcement: using Bayes risk for active learning in POMDPs (2008, neutral): “To date, POMDP-based planners have been us d for applications as diverse as robot navigation [1, 2], ecological monitorin g [3], dynamic pricing [4], and several areas of dialog management and healthca re [5, 6, 7].”
Guided Dialog Policy Learning: Reward Estimation for Multi-Domain Task-Oriented Dialog (2019, background): “Given a user action, the belief state of its slot type is updated according to its slot value (Roy et al., 2000).”