Many-Goals Reinforcement Learning

By Vivek Veeriah, Junhyuk Oh, Satinder Singh. 2018

56 citations recorded by the source corpus.

All-goals updating exploits the off-policy nature of Q-learning to update all possible goals an agent could have from each transition in the world, and was introduced into Reinforcement Learning (RL) by Kaelbling (1993). In prior work this was mostly explored in small-state RL problems that allowed tabular representations and where all possible goals could be explicitly enumerated and learned separately. In this paper we empirically explore 3 different extensions of the idea of updating many (instead of all) goals in the context of RL with deep neural networks (or DeepRL for short). First, in a direct adaptation of Kaelbling's approach we explore if many-goals updating can be used to achieve mastery in non-tabular visual-observation domains. Second, we explore whether many-goals updating can be used to pre-train a network to subsequently learn faster and better on a single main task of interest. Third, we explore whether many-goals updating can be used to provide auxiliary task updates in training a network to learn faster and better on a single main task of interest. We provide comparisons to baselines for each of the 3 extensions.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Electrical and Electronic Engineering

Citations in the corpus

Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “Veeriah, Oh, and Singh (2018) propose a deep RL adaptation of the work of Kaelbling (1993).”
CURIOUS: Intrinsically Motivated Modular Multi-Goal Reinforcement Learning (2018, unknown): “Veeriah et al. (2018) uses LP computed from Bellman errors for goal selection, but this form of LP does not improve over random goal selection.”
Skew-Fit: State-Covering Self-Supervised Reinforcement Learning (2019, methodology): “Some prior methods propose to choose goals based on heuristics such as learning progress (Baranes & Oudeyer, 2012; Veeriah et al., 2018; Colas et al., 2018a), how offpolicy the goal is (Nachum et al., 2018), or level of difficulty (Florensa et al., 2018b).”
Unsupervised Control Through Non-Parametric Discriminative Rewards (2018, background): “Veeriah et al. (2018) expands upon hindsight replay and the all-goal update strategy proposed by Kaelbling (1993), generalizing the latter to non-tabular environments and exploring related strategies for skill discovery, unsupervised pre-training and auxiliary tasks.”
Planning with Goal-Conditioned Policies (2019, background): “Goal-conditioned reinforcement learning has been studied in a number of prior works [24, 25, 37, 18, 53, 2, 48, 57, 40, 59].”
Hyperbolic Discounting and Learning over Multiple Horizons (2019, background): “…the low-information coming from the environment through extrinsic rewards (Lample & Chaplot, 2017; Mirowski et al., 2016), (Jaderberg et al., 2016; Veeriah et al., 2018; Sutton et al., 2011) 3 B ELIEF OF R ISK I MPLIES A D ISCOUNT F UNCTION Sozou (1998) formalizes time preferences in which future…”
Discovery of Useful Questions as Auxiliary Tasks (2019, background): “…may also drive the acquisition of a diverse set of skills, even before the main task is known, forming a basis for policy composition or exploration (Andrychowicz et al., 2016; Veeriah et al., 2018; Eysenbach et al., 2018; Florensa et al., 2018; Mankowitz et al., 2018; Riedmiller et al., 2018).”