Discovery of Useful Questions as Auxiliary Tasks

By Vivek Veeriah, Matteo Hessel, Zhongwen Xu, Richard L. Lewis, Janarthanan Rajendran, Junhyuk Oh, Hado van Hasselt, David Silver, Satinder Singh. 2019

90 citations recorded by the source corpus.

Arguably, intelligent agents ought to be able to discover their own questions so that in learning answers for them they learn unanticipated useful knowledge and skills; this departs from the focus in much of machine learning on agents learning answers to externally defined questions. We present a novel method for a reinforcement learning (RL) agent to discover questions formulated as general value functions or GVFs, a fairly rich form of knowledge representation. Specifically, our method uses non-myopic meta-gradients to learn GVF-questions such that learning answers to them, as an auxiliary task, induces useful representations for the main task faced by the RL agent. We demonstrate that auxiliary tasks based on the discovered GVFs are sufficient, on their own, to build representations that support main task learning, and that they do so better than popular hand-designed auxiliary tasks from the literature. Furthermore, we show, in the context of Atari2600 videogames, how such auxiliary tasks, meta-learned alongside the main task, can improve the data efficiency of an actor-critic agent.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “Recent work (Veeriah et al., 2019) addresses discovering useful auxiliary tasks in the form of the question functions of a General Value Function (GVF) (Sutton et al.”
Meta-Learning in Neural Networks: A Survey (2020, background): “preform meta-optimization online within a single base learning episode [42], [180], [197], [198].”
Meta-Gradient Reinforcement Learning with an Objective Discovered Online (2020, unknown): “…optimiser RL 2 , Meta-RL [9, 39] X recurrent network MAML, REPTILE [11, 23] ← initial params Meta-Gradient [43, 46] → γ , λ , reward Meta-Gradient [38, 44, 40] ← auxiliary tasks, hyperparams, reward weights ML 3 , MetaGenRL [2, 19] ← loss function Evolved PG [16] X loss function Oh et al. 2020…”
Self-Tuning Deep Reinforcement Learning (2020, methodology): “In previous work, online metagradients have been used to learn the discount factor or the λ coefficient (Xu et al., 2018), to discover intrinsic rewards (Zheng et al., 2018) and auxiliary tasks (Veeriah et al., 2019).”
Discovering Reinforcement Learning Algorithms (2020, methodology): “This approach has been applied to meta-learn various forms of algorithmic components such as the discount factor [38], intrinsic rewards [41], auxiliary tasks [33], returns [36], auxiliary policy updates [42], off-policy corrections [39], and update target [37].”
What Can Learned Intrinsic Rewards Capture? (2019, background): “, 2018b), auxiliary tasks (Schlegel et al., 2018; Veeriah et al., 2019), unsupervised learning rules (Metz et al.”
Online Meta-Critic Learning for Off-Policy Actor-Critic Methods (2020, background): “, 2018), hyperparameters (Xu et al., 2018; Veeriah et al., 2019), and intrinsic rewards (Zheng et al.”
Vision-Language Navigation With Self-Supervised Auxiliary Reasoning Tasks (2019, background): “Moreover, the concept of learning from auxiliary tasks to improve data efficiency and robustness [16, 28, 39, 23] has been extensively investigated in reinforcement learning.”
Meta-learning curiosity algorithms (2020, methodology): “Most work on meta-RL has focused on learning transferable feature representations or parameter values for quickly adapting to new tasks (Finn et al., 2017; Finn, 2018; Clavera et al., 2019) or improving performance on a single task (Xu et al., 2018; Veeriah et al., 2019).”