Efficient exploration with Double Uncertain Value Networks

By Thomas M. Moerland, Joost Broekens, C. Jonker. 2017

48 citations recorded by the source corpus.

This paper studies directed exploration for reinforcement learning agents by tracking uncertainty about the value of each available action. We identify two sources of uncertainty that are relevant for exploration. The first originates from limited data (parametric uncertainty), while the second originates from the distribution of the returns (return uncertainty). We identify methods to learn these distributions with deep neural networks, where we estimate parametric uncertainty with Bayesian drop-out, while return uncertainty is propagated through the Bellman equation as a Gaussian distribution. Then, we identify that both can be jointly estimated in one network, which we call the Double Uncertain Value Network. The policy is directly derived from the learned distributions based on Thompson sampling. Experimental results show that both types of uncertainty may vastly improve learning in domains with a strong exploration challenge.

Subfields: Artificial Intelligence, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

Citations in the corpus

The Uncertainty Bellman Equation and Exploration (2017, background): “We show that, just as the Bellman equation relates the value of a policy beyond a single time-step, so too does the uncertainty Bellman equation propagate uncertainty values over multiple time-steps, thereby facilitating ‘deep exploration’ (Osband et al., 2017; Moerland et al., 2017).”
Implicit Quantile Networks for Distributional Reinforcement Learning (2018, background): “The choice between them amounts to a choice 1 One exception is the recent work (Moerland et al., 2017) towards combining both forms of uncertainty to improve exploration. over whether the behavior should be invariant to mixing with random events or to convex combinations of outcomes.”
QUOTA: The Quantile Option Architecture for Reinforcement Learning (2018, background): “Recently, Moerland, Broekens, and Jonker (2017) combined the two uncertainty via the Double Uncertain Value Network and demonstrated performance boost in simple domains.”