DORA The Explorer: Directed Outreaching Reinforcement Action-Selection

By Leshem Choshen, Lior Fox, Yonatan Loewenstein. 2018

70 citations recorded by the source corpus.

Exploration is a fundamental aspect of Reinforcement Learning, typically implemented using stochastic action-selection. Exploration, however, can be more efficient if directed toward gaining new world knowledge. Visit-counters have been proven useful both in practice and in theory for directed exploration. However, a major limitation of counters is their locality. While there are a few model-based solutions to this shortcoming, a model-free approach is still missing. We propose $E$-values, a generalization of counters that can be used to evaluate the propagating exploratory value over state-action trajectories. We compare our approach to commonly used RL techniques, and show that using $E$-values improves learning and performance over traditional counters. We also show how our method can be implemented with function approximation to efficiently learn continuous MDPs. We demonstrate this by showing that our approach surpasses state of the art performance in the Freeway Atari 2600 game.

Subfields: Computer Science Applications, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Science Applications

In Reinforcement Learning

Citations in the corpus

Safe Policy Improvement with Soft Baseline Bootstrapping (2019, methodology): “We recall that the computation of the policy improvement step relies on the estimates of an error function eP , which may, for instance, be indirectly inferred from pseudo-counts ÑD(x, a) (Bellemare et al., 2016; Fox et al., 2018; Burda et al., 2019).”
Safe Policy Improvement with an Estimated Baseline Policy (2019, methodology): “[1, 2, 5] investigated some methods for optimism-based exploration, which proved to not be robust enough for pessimism based purpose, where there is a requirement for exhaustiveness.”
Exploration by Random Network Distillation (2018, background): “…are a natural and effective way to do exploration (Strehl & Littman, 2008) and a lot of work has studied how to tractably generalize count bonuses to large state spaces (Bellemare et al., 2016; Fu et al., 2017; Ostrovski et al., 2017; Tang et al., 2017; Machado et al., 2018; Fox et al., 2018).”