Deep Exploration via Randomized Value Functions

By Ian Osband, Daniel Russo, Zheng Wen, Benjamin Van Roy. 2017

339 citations recorded by the source corpus.

We study the use of randomized value functions to guide deep exploration in reinforcement learning. This offers an elegant means for synthesizing statistically and computationally efficient exploration with common practical approaches to value function learning. We present several reinforcement learning algorithms that leverage randomized value functions and demonstrate their efficacy through computational studies. We also prove a regret bound that establishes statistical efficiency with a tabular representation.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

The Uncertainty Bellman Equation and Exploration (2017, background): “Due to this difficulty prior work has sought to approximate the posterior distribution (Osband et al., 2017), and use that to drive exploration.”
Why is Posterior Sampling Better than Optimism for Reinforcement Learning? (2016, methodology): “lysis will be to show that the contribution from the transition estimate P H h=1 jw P(x kh)j concentrates at a rate independent of S. At the root of our argument is the notion of stochastic optimism (Osband, 2016), which introduces a partial ordering over random variables. We make particular use of ”
Variational Bayesian Reinforcement Learning with Regret Bounds (2018, methodology): “We consider a small tabular MDP called DeepSea [39] shown in Figure 1, which can be thought of as an unrolled version of the RiverSwim environment [52].”
MAVEN: Multi-Agent Variational Exploration (2019, background): “[37] extends the Bootstrapped DQN to include a prior.”
Noisy Networks for Exploration (2017, background): “At a high level our algorithm induces a randomised network for exploration, with care this approach can be provably-efficient with suitable linear value functions [24].”
Behaviour Suite for Reinforcement Learning (2019, background): “Reinforcement learning calls for a sophisticated form of exploration called deep exploration [44, 49].”
Randomized Prior Functions for Deep Reinforcement Learning (2018, background): “This can lead to arbitrarily poor decision making, even when combined with an e cient strategy for exploration [45].”
Playing hard exploration games by watching YouTube (2018, background): “First, intrinsic motivation methods provide an auxiliary reward that encourages the agent to explore states or action trajectories that are “novel” or “informative” with respect to some measure [8, 30, 27].”
Efficient Exploration Through Bayesian Deep Q-Networks (2018, background): “To remedy these failings [32] consider the use of randomized value functions to approximate posterior samples for the value function in a computationally efficient manner.”
A Tutorial on Thompson Sampling (2017, background): “Learning about the true dynamics of the MDP requires a consistent policy over N steps right or N steps left; a variant of Thompson sampling that resampled every timestep would be exponentially unlikely to make it to either end within N steps [34].”
Bayesian Reinforcement Learning in Factored POMDPs (2018, background): “These methods often require millions of samples and combining them with effective exploration, although a topic of some studies [3, 5, 33], generally is difficult.”
ReLMoGen: Leveraging Motion Generation in Reinforcement Learning for Mobile Manipulation (2020, background): “Often, the steps in free space do not return any reward, rendering mobile manipulation hard exploration problems [12, 13].”
ReLMoGen: Integrating Motion Generation in Reinforcement Learning for Mobile Manipulation (2021, methodology): “Often, the steps in free space do not return any reward, rendering mobile manipulation hard exploration problems [12, 13].”
Offline Policy Selection under Uncertainty (2020, background): “However, there are several signiﬁcant diﬀerences: i), the ﬁrst and most diﬀerent is the modeling object, Osband et al. 2018) considers ﬁxed ﬁnite-horizon case.”
BeBold: Exploration Beyond the Boundary of Explored Regions (2020, background): “Outside of intrinsic reward, researchers have proposed to use randomized value functions to encourage exploration (Osband et al., 2016; Hessel et al., 2017; Osband et al., 2019).”
Randomized Value Functions via Multiplicative Normalizing Flows (2018, background): “This naive perturbation is not directed; it continuously explores actions that are known to be sub-optimal and may result in sample complexity that grows exponentially with the number of states (Kearns and Singh, 2002; Osband et al., 2017).”
Statistical Bootstrapping for Uncertainty Estimation in Off-Policy Evaluation (2020, methodology): “E, the ideas behind Efron’s bootstrap have inspired a number of existing RL algorithms. Speciﬁcally, statistical bootstrapping has been proposed as a mechanism for exploration; e.g., bootstrapped DQN [26, 27]. However, in practice, the type of bootstrapping performed in these algorithms is far from ”
Exploration in Deep Reinforcement Learning: A Survey (2022, background): “[113] LSVI Optimistic Cartpole Swing up: 600 (DQN 0) State Id Ac MF D/ D Nikolov et al.”