BBQ-Networks: Efficient Exploration in Deep Reinforcement Learning for Task-Oriented Dialogue Systems
By Zachary Chase Lipton, Xiujun Li, Jianfeng Gao, Lihong Li, Faisal Ahmed, Li Deng. 2016
173 citations recorded by the source corpus.
We present a new algorithm that significantly improves the efficiency of exploration for deep Q-learning agents in dialogue systems. Our agents explore via Thompson sampling, drawing Monte Carlo samples from a Bayes-by-Backprop neural network. Our algorithm learns much faster than common exploration strategies such as ε-greedy, Boltzmann, bootstrapping, and intrinsic-reward-based ones. Additionally, we show that spiking the replay buffer with experiences from just a few successful episodes can make Q-learning feasible when it might otherwise fail.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning, Dialogue & Conversational AI
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Yoshua Bengio
- Pieter Abbeel
- Jianfeng Gao
- Mohit Bansal
- Ruslan Salakhutdinov
- Percy Liang
- Quoc V. Le
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Andrew Zisserman
- Jitendra Malik
- Andrea Vedaldi
In Reinforcement Learning
- Sergey Levine
- Pieter Abbeel
- Yuxi Li
- Emma Brunskill
- Marc G. Bellemare
- R´emi Munos
- Joelle Pineau
- Hado van Hasselt
In Dialogue & Conversational AI
- Minlie Huang
- Charles Blundell
- Anima Anandkumar
- Emma Brunskill
- Kamyar Azizzadenesheli
- Meire Fortunato
- Xiaoyan Zhu
- Oriol Vinyals
Citations in the corpus
- Observe and Look Further: Achieving Consistent Performance on Atari (2018, methodology): “[12] Zachary Lipton, Xiujun Li, Jianfeng Gao, Lihong Li, Faisal Ahmed, and Li Deng.”
- Neural Approaches to Conversational AI (2018, neutral): “Many published studies involving human subjects are of the first type (Walker, 2000; Singh et al., 2002; Ai et al., 2007; Rieser and Lemon, 2011; Gašić et al., 2013; Wen et al., 2015; Young et al., 2016; Peng et al., 2017; Lipton et al., 2018).”
- Dynamic Dialogue Policy for Continual Reinforcement Learning (2022, background): “The approaches directly vectorise the belief state, hence the size of the input vector depends on the domain (as different domains for instance have different numbers of slots) (Su et al., 2016; Lipton et al., 2018; Weisz et al., 2018; Takanobu et al., 2019; Zhu et al., 2020).”
- Incremental Learning from Scratch for Task-Oriented Dialogue Systems (2019, methodology): “Lipton et al. (2018) proposed to use BBQ-Networks to extend the domain.”
- Noisy Networks for Exploration (2017, background): “Consequently, it also differs from Thompson sampling (Thompson, 1933; Lipton et al., 2016) as the distribution on the parameters of our agents does not necessarily converge to an approximation of a posterior distribution.”
- Efficient Exploration Through Bayesian Deep Q-Networks (2018, methodology): “Other works suggest using a variational approximation to the Q-networks (Lipton et al., 2016b) or a concurrent work on noisy network (Fortunato et al., 2017).”
- Challenges in Building Intelligent Open-domain Dialog Systems (2019, background): “3Dialog management performs both dialog state tracking [47, 89] and response selection via policy [71, 99, 132, 183].”
- Modeling Multi-turn Conversation with Deep Utterance Aggregation (2018, background): “Towards this end, a number of data-driven dialogue systems are designed (Lowe et al., 2015; Wu et al., 2017; Wen et al., 2017; Mei et al., 2017; Young et al., 2018; Lipton et al., 2018), in which modeling multi-turn conversation has drawn more and more attention.”
- A Teacher-Student Framework for Maintainable Dialog Manager (2018, methodology): “Lipton et al. (2017) proposed to use BBQ-Networks to extend the domain.”
- Hybrid Reinforcement Learning with Expert State Sequences (2019, background): “Gilbert et al. (2015), Lipton et al. (2016) and Lakshminarayanan, Ozair, and Ben-gio (2016) store expert state-action pairs in a replay buffer to facilitate learning.”
- Dynamic Dialogue Policy Transformer for Continual Reinforcement Learning (2022, background): “The approaches directly vec-torise the belief state, hence the size of the input vector depends on the domain (as different domains for instance have different numbers of slots) (Su et al., 2016; Lipton et al., 2018; Weisz et al., 2018; Takanobu et al., 2019; Zhu et al., 2020).”
- Randomized Value Functions via Multiplicative Normalizing Flows (2018, methodology): “MNF-DQN and BBQN have an extra hyperparameter λ, the trade-off parameter between the log likelihood cost and the regularization cost.”
- Bayesian Recurrent Neural Networks (2017, methodology): “…trained networks has been used successfully for training feedfor-ward models for supervised learning and to aid exploration by reinforcement learning agents (Blundell et al., 2015; Lipton et al., 2016; Houthooft et al., 2016), but as yet, it has not been applied to recurrent neural networks.”
- What Should I Ask? Using Conversationally Informative Rewards for Goal-oriented Visual Dialog. (2019, unknown): “(Lipton et al., 2017), who used information gain and Q-learning to generate goal-oriented questions for movie recommendations.”