Ready Policy One: World Building Through Active Learning

By Philip J. Ball, Jack Parker-Holder, Aldo Pacchiano, Krzysztof Marcin Choromanski, Stephen J. Roberts. 2020

54 citations recorded by the source corpus.

Model-Based Reinforcement Learning (MBRL) offers a promising direction for sample efficient learning, often achieving state of the art results for continuous control tasks. However, many existing MBRL methods rely on combining greedy policies with exploration heuristics, and even those which utilize principled exploration bonuses construct dual objectives in an ad hoc fashion. In this paper we introduce Ready Policy One (RP1), a framework that views MBRL as an active learning problem, where we aim to improve the world model in the fewest samples possible. RP1 achieves this by utilizing a hybrid objective function, which crucially adapts during optimization, allowing the algorithm to trade off reward v.s. exploration at different stages of learning. In addition, we introduce a principled mechanism to terminate sample collection once we have a rich enough trajectory batch to improve the model. We rigorously evaluate our method on a variety of continuous control tasks, and demonstrate statistically significant gains over existing approaches.

Subfields: Artificial Intelligence, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

Citations in the corpus

Same State, Different Task: Continual Reinforcement Learning without Interference (2021, methodology): “We use a modified version of the Exponentially Weighted Average Forecaster algorithm (Cesa-Bianchi and Lugosi 2006), as has been shown to be successful adapting components of RL algorithms (Ball et al. 2020).”
Planning to Explore via Self-Supervised World Models (2020, background): “Concurrently with us, (Ball et al., 2020) discusses the connection between information gain and model disagreement in the context of task-speciﬁc exploration from low-dimensional state information.”
Effective Diversity in Population-Based Reinforcement Learning (2020, methodology): “…this adaptive mechanism could also be used for count-based exploration methods or intrinsic rewards (Schmidhuber, 2010), and note very recent work using a similar approach to vary exploration in off-policy methods (Schaul et al., 2019; Badia et al., 2020) and model-based RL (Ball et al., 2020).”