Efﬁcient Exploration for Dialogue Policy Learning with BBQ Networks & Replay Buffer Spiking

By Zachary Chase Lipton, Jianfeng Gao, Lihong Li, Xiujun Li, Faisal Ahmed, Li Deng. 2016

Subfields: Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Surprising Negative Results for Generative Adversarial Tree Search (2018, background): “For example (Cuayáhuitl, 2016; Fatemi et al., 2016; Wen et al., 2016) investigate DRL for dialogue policy learning, with (Lipton et al., 2018) addressing the efﬁciency of exploration.”
A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inference (2019, methodology): “The uncertainty afforded by Bayes by Backprop trained neural networks has been used successfully for training feedforward neural networks in both supervised and reinforcement learning environments [4, 42, 28], for training recurrent neural networks [11], but has not been applied to convolutional neu”
Variational Bayesian Reinforcement Learning with Regret Bounds (2018, background): “Approximations to the optimal Bayesian policy exist, one of the most successful being Thompson sampling [53, 55] wherein the agent samples from the posterior over value functions and acts greedily with respect to that sample [38, 42, 27, 41].”
Noisy Networks for Exploration (2017, background): “It also differs from variational inference [11, 8] or Thompson sampling [35, 19] as the distribution on the parameters of our agents does not necessarily converge to an approximation of a posterior distribution.”
Randomized Prior Functions for Deep Reinforcement Learning (2018, background): “Training V (s1)=X according to the loss of [16, 33] will lead to zero uncertainty estimates at s1, when in fact V (s1)∼N(μY ,σ2 Y ).”
Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation (2018, background): “Curiosity or uncertainty has been used as a signal for exploration [33, 35, 23, 29].”
Efficient Exploration Through Bayesian Deep Q-Networks (2018, methodology): “Other works suggest using a variational approximation to the Q-networks (Lipton et al., 2016b) or a concurrent work on noisy network (Fortunato et al., 2017).”
Hybrid Reinforcement Learning with Expert State Sequences (2019, background): “Gilbert et al. (2015), Lipton et al. (2016) and Lakshminarayanan, Ozair, and Ben-gio (2016) store expert state-action pairs in a replay buffer to facilitate learning.”
Sample Efficient Deep Reinforcement Learning for Dialogue Systems With Large Action Spaces (2018, methodology): “Uncertainty estimates can be incorporated into DQN using Bayes-by-Backprop [25].”
Distributed Structured Actor-Critic Reinforcement Learning for Universal Dialogue Management (2020, background): “the past few years, there are many deep reinforcement learning (DRL) algorithms which use neural networks (NN) as function approximators, investigated for dialogue policy [2], [3], [4], [5], [6].”
Bayesian Convolutional Neural Networks with Variational Inference (2018, methodology): “The uncertainty afforded by Bayes by Backprop trained neural networks has been used successfully for training feedforward neural networks in both supervised and reinforcement learning environments [1, 22, 13], for training recurrent neural networks [5], but has not been applied to convolutional neur”
An Introduction to Deep Reinforcement Learning (2018, unknown): “Along similar lines, efficient exploration has been obtained by the induced stochasticity of uncertainty estimates given by a dropout Q-network (Gal and Ghahramani, 2016) or parametric noise added to its weights (Lipton et al., 2016; Plappert et al., 2017; Fortunato et al., 2017).”
Combating Reinforcement Learning's Sisyphean Curse with Intrinsic Fear (2016, background): “Some investigated applications include robotics [13], dialogue systems [7, 15], energy management [21], and selfdriving cars [23].”
Bayesian Convolutional Neural Networks (2018, unknown): “Denker and Lecun (1991) and MacKay (1995) investigated using the Laplace approximation for capturing the posterior of neural networks.”
Combating Deep Reinforcement Learning ’ s Sisyphean Curse with Reinforcement Learning (2017, background): “At such time, our networks are subject to the classic problem of catastrophic interference McCloskey and Cohen [1989], McClelland et al. [1995]. Nothing prevents the DQN’s policy from drifting back towards one that revisits catastrophic, but long-forgotten mistakes.”
Bayesian Recurrent Neural Networks (2017, methodology): “…trained networks has been used successfully for training feedfor-ward models for supervised learning and to aid exploration by reinforcement learning agents (Blundell et al., 2015; Lipton et al., 2016; Houthooft et al., 2016), but as yet, it has not been applied to recurrent neural networks.”
Sample-Efficient Deep RL with Generative Adversarial Tree Search (2018, background): “, 2016) investigate DRL for dialogue policy learning, with (Lipton et al., 2018) addressing the efficiency of exploration.”
Deep Reinforcement Learning: An Overview (2017, unknown): “…papers follow: Asri et al. (2016), Bordes and Weston (2016), Chen et al. (2016b), Dhingra et al. (2016), Fatemi et al. (2016), Li et al. (2016a), Lipton et al. (2016), Mesnil et al. (2015), Mo et al. (2016), Shah et al. (2016), Su et al. (2016a), Wen et al. (2015a), Williams and Zweig (2016),…”