Sample-efficient batch reinforcement learning for dialogue management optimization

By Olivier Pietquin, Matthieu Geist, Senthilkumar Chandramohan, Hervé Frezza-Buet. 2011

109 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Social Psychology, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Social Psychology

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

D4RL: Datasets for Deep Data-Driven Reinforcement Learning (2020, methodology): “Offline reinforcement learning using large datasets has also been used in real-world systems where evaluation using a simulator is not possible, such as in robotics [Cabi et al., 2019] and dialogue systems [Henderson et al., 2008, Pietquin et al., 2011, Jaques et al., 2019].”
Safe Policy Improvement with Baseline Bootstrapping (2017, background): “This scenario is typically encountered when a policy is trained in a simulator and then run in its real environment, for instance in Transfer RL (Taylor & Stone, 2009); or when a system is designed with expert knowledge and then optimized, for example in Dialogue applications (Pietquin et al., 2011;”
End-to-End Offline Goal-Oriented Dialog Policy Learning via Policy Gradient (2017, background): “Some batch reinforcement learning algorithms have been proposed [16, 11].”
RL Unplugged: Benchmarks for Offline Reinforcement Learning (2020, background): “…domains such as robots (Cabi et al., 2019; Gu et al., 2017; Kalashnikov et al., 2018) or dialog (Henderson et al., 2008; Jaques et al., 2019; Pietquin et al., 2011), but it is often difficult to do thorough evaluations in these domains for the same reason offline RL is useful in them,…”
Deal or No Deal? End-to-End Learning of Negotiation Dialogues (2017, methodology): “RL has been widely used to improve dialogue managers, which manage transitions between dialogue states (Singh et al., 2002; Pietquin et al., 2011; Rieser and Lemon, 2011; Gaˇsic et al., 2013; Fatemi et al., 2016).”
Benchmarking Batch Deep Reinforcement Learning Algorithms (2019, background): “Learning with ﬁnite data sets is invaluable for a variety of real-world applications, where data collection may be difﬁcult, time-consuming or costly [Guez et al., 2008, Pietquin et al., 2011, Gauci et al., 2018].”
Batch Policy Learning under Constraints (2019, background): “FQI has been shown to work well with several empirical domains: spoken dialogue systems (Pietquin et al., 2011), physical robotic soccer (Riedmiller et al.”
Frames: a corpus for adding memory to goal-oriented dialogue systems (2017, background): “Performing more than one action per turn is a challenge when using reinforcement learning (Pietquin et al., 2011; Gašić et al., 2012; Fatemi et al., 2016) and, to our knowledge, has only been tackled in a simulated setting (Laroche et al.”
End-to-end optimization of goal-driven and visually grounded dialogue systems (2017, neutral): “On the other hand, RL approaches could handle the planning and the non-differentiable metric problems but require online learning (although batch learning is possible but dif-ﬁcult with low amounts of data [ Pietquin et al. , 2011 ] ).”
How to Motivate Your Dragon: Teaching Goal-Driven Agents to Speak and Act in Fantasy Worlds (2020, background): “RL has been studied for such tasks, usually to improve dialogue state management (Singh et al., 2000; Pietquin et al., 2011; Fatemi et al., 2016) and to improve response quality (Li et al.”
Datasets for Data-Driven Reinforcement Learning (2020, methodology): “Offline reinforcement learning using large datasets has also been used in real-world systems where evaluation using a simulator is not possible, such as in robotics [Cabi et al., 2019] and dialogue systems [Henderson et al., 2008, Pietquin et al., 2011, Jaques et al., 2019].”
I love your chain mail! Making knights smile in a fantasy game world: Open-domain goal-oriented dialogue agents (2019, background): “Typically, they used RL to improve dialogue managers, which manage transitions between dialogue states (Singh et al., 2002; Pietquin et al., 2011; Rieser & Lemon, 2011; Gasic et al., 2013; Fatemi et al., 2016).”
POMDP-Based Statistical Spoken Dialog Systems: A Review (2013, methodology): “done to automate feature selection [43], [44].”
Towards Solving Text-based Games by Producing Adaptive Action Spaces (2018, background): “The task of choosing the best next utterance to generate for a given context has been extensively studied in the literature on dialogue systems (Rieser and Lemon, 2016; Pietquin et al., 2011; Fatemi et al., 2016).”
Data-Driven Methods for Adaptive Spoken Dialogue Systems (2012, methodology): “For example, previous research systems, such as TrindiKit [21] and DIPPER [5], implemented these rules using the “Information State Update” approach to dialogue management, whereas in industry, standards such as Voice XML1 are common practice [27].”
MAKING KNIGHTS SMILE IN A FANTASY GAME WORLD (2019, background): “Typically, they used RL to improve dialogue managers, which manage transitions between dialogue states (Singh et al., 2002; Pietquin et al., 2011; Rieser & Lemon, 2011; Gasic et al., 2013; Fatemi et al., 2016).”
Situated Dialogue Learning through Procedural Environment Generation (2021, methodology): “Goal oriented Dialogue Sub-tasks within the overall task of goal oriented dialogue, such as dialogue state management (Singh et al., 2000; Pietquin et al., 2011; Fatemi et al., 2016) and response generation (Li et al., 2016) have used RL to boost performance.”
Offline Reinforcement Learning with Pseudometric Learning (2021, background): “Offline RL (Lagoudakis & Parr, 2003; Ernst et al., 2005; Riedmiller, 2005; Pietquin et al., 2011; Lange et al.; Levine et al., 2020) methods suffer from overestimation of state-action pairs that are not in the support of logged transitions.”
Domain-independent User Simulation with Transformers for Task-oriented Dialogue Systems (2021, background): “The system typically needs thousands of interactions to train a usable policy (Schatzmann et al., 2007; Pietquin et al., 2011; Li et al., 2016; Shi et al., 2019).”