Efficient Exploration for Dialog Policy Learning with Deep BBQ Networks \& Replay Buffer Spiking

By Zachary Chase Lipton, Jianfeng Gao, Lihong Li, Xiujun Li, Faisal Ahmed, Li Deng. 2016

58 citations recorded by the source corpus.

Subfields: Reinforcement Learning, Dialogue & Conversational AI

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Dialogue & Conversational AI

Citations in the corpus

Surprising Negative Results for Generative Adversarial Tree Search (2018, background): “However, in real-world applications, such as robotics (Levine et al., 2016) and dialogue systems (Lipton et al., 2016), collecting samples often takes considerable time and effort.”
Deep Q-learning From Demonstrations (2017, methodology): “The right subplots in Figure 2 compare DQfD with three related algorithms for leveraging demonstration data in DQN: • Replay Buffer Spiking (RBS) (Lipton et al. 2016) • Human Experience Replay (HER) (Hosu and Rebedea 2016) • Accelerated DQN with Expert Trajectories (ADET) (Lak- shminarayanan, Ozair,”
Learning from Demonstrations for Real World Reinforcement Learning (2017, methodology): “(Lipton et al., 2016) show promising results with initializing the DQN agent’s replay buffer with demonstration data on dialog tasks, but they do not pre-train the agent for good initial performance.”
Noisy Networks for Exploration (2017, background): “Consequently, it also differs from Thompson sampling (Thompson, 1933; Lipton et al., 2016) as the distribution on the parameters of our agents does not necessarily converge to an approximation of a posterior distribution.”
Efficient Exploration Through Bayesian Deep Q-Networks (2018, methodology): “Other works suggest using a variational approximation to the Q-networks (Lipton et al., 2016b) or a concurrent work on noisy network (Fortunato et al., 2017).”
Bayesian Convolutional Neural Networks (2018, methodology): “…been used successfully for training feedforward neural networks in both supervised and reinforcement learning environments (Blundell et al. 2015, Lipton et al. 2016, Houthooft et al. 2016), for training recurrent neural networks (Fortunato, Blundell, and Vinyals 2017), but has not been applied…”
Learn What Not to Learn: Action Elimination with Deep Reinforcement Learning (2018, background): “Lipton et al. (2016a) studied a related problem where an agent wants to avoid catastrophic forgetting of dangerous states.”
A review of cooperative multi-agent deep reinforcement learning (2019, methodology): “To address this issue, softmax operator and Boltzmann softmax operator are added to DQN [116, 152] to get the probability of choosing each action,”
Bayesian Recurrent Neural Networks (2017, methodology): “…trained networks has been used successfully for training feedfor-ward models for supervised learning and to aid exploration by reinforcement learning agents (Blundell et al., 2015; Lipton et al., 2016; Houthooft et al., 2016), but as yet, it has not been applied to recurrent neural networks.”
Randomized Value Functions via Multiplicative Normalizing Flows (2018, methodology): “MNF-DQN and BBQN have an extra hyperparameter λ, the trade-off parameter between the log likelihood cost and the regularization cost.”
Generating Adversarial Driving Scenarios in High-Fidelity Simulators (2019, methodology): “Methods in this category have modeled the uncertainty in the value function using the bootstrap method [14], Bayesian linear regression [15], and variational inference [16].”