Meta-learning curiosity algorithms

By Ferran Alet, Martin Schneider, Tomas Lozano-Perez, Leslie Pack Kaelbling. 2020

67 citations recorded by the source corpus.

Exploration is a key component of successful reinforcement learning, but optimal approaches are computationally intractable, so researchers have focused on hand-designing mechanisms based on exploration bonuses and intrinsic reward, some inspired by curious behavior in natural systems. In this work, we propose a strategy for encoding curiosity algorithms as programs in a domain-specific language and searching, during a meta-learning phase, for algorithms that enable RL agents to perform well in new domains. Our rich language of programs, which can combine neural networks with other building blocks including nearest-neighbor modules and can choose its own loss functions, enables the expression of highly generalizable programs that perform well in domains as disparate as grid navigation with image input, acrobot, lunar lander, ant and hopper. To make this approach feasible, we develop several pruning techniques, including learning to predict a program's success based on its syntactic properties. We demonstrate the effectiveness of the approach empirically, finding curiosity strategies that are similar to those in published literature, as well as novel strategies that are competitive with them and generalize well.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

AutoML-Zero: Evolving Machine Learning Algorithms From Scratch (2020, methodology): “…rule used during backpropagation (Andrychowicz et al., 2016; Ravi & Larochelle, 2017), the data augmentation (Cubuk et al., 2019a; Park et al., 2019) or the intrinsic curiosity reward in reinforcement learning (Alet et al., 2019); in these works, all other algorithmic aspects remain hand-designed.”
Meta-Learning in Neural Networks: A Survey (2020, background): “Successful applications have been demonstrated in areas spanning few-shot image recognition [19], [23], unsupervised learning [16], data efficient [24], [25] and self-directed [26] reinforcement learning (RL), hyperparameter optimization [20], and neural architecture search (NAS) [21], [27], [28].”
Improving Generalization in Meta Reinforcement Learning using Learned Objectives (2019, methodology): “Concurrent to our work, Alet et al. (2020) uses techniques from architecture search to search for viable artiﬁcial curiosity objectives that are composed of primitive objective functions.”
Evolving Reinforcement Learning Algorithms (2021, methodology): “Another approach searches over the space of curiosity programs using a similar language of DAGs with neural network modules (Alet et al., 2020a) and performs the meta-training on a single environment.”
Bootstrapped Meta-Learning (2021, background): “More generally, the update rule can be an arbitrary parameterised function (Hochreiter et al., 2001; Andrychowicz et al., 2016; Kirsch et al., 2019; Oh et al., 2020), or the function itself can be meta-learned jointly with its parameters (Alet et al., 2020; Real et al., 2020).”
Introducing Symmetries to Black Box Meta Reinforcement Learning (2021, background): “Recent work in meta reinforcement learning (RL) has begun to tackle the challenging problem of automatically discovering general-purpose RL algorithms (Kirsch, van Steenkiste, and Schmidhuber 2020; Alet et al. 2020; Oh et al. 2020).”
Discovering Reinforcement Learning Algorithms (2020, background): “Discovering Reinforcement Learning Algorithms There have been a few attempts to meta-learn RL algorithms, from earlier work on bandit algorithms [20, 19] to curiosity algorithms [1] and RL objectives [16, 40, 6, 17] (see Table 1 for comparison).”