Online meta-learning by parallel algorithm competition
By Stefan Elfwing, Eiji Uchibe, Kenji Doya. 2017
25 citations recorded by the source corpus.
The efficiency of reinforcement learning algorithms depends critically on a few meta-parameters that modulate the learning updates and the trade-off between exploration and exploitation. The adaptation of the meta-parameters is an open question, which arguably has become a more important issue recently with the success of deep reinforcement learning. The long learning times in domains such as Atari 2600 video games makes it not feasible to perform comprehensive searches of appropriate meta-parameter values. In this study, we propose the Online Meta-learning by Parallel Algorithm Competition (OMPAC) method, which is a novel Lamarckian evolutionary approach to online meta-parameter adaptation. The population consists of several instances of a reinforcement learning algorithm which are run in parallel with small differences in initial meta-parameter values. After a fixed number of learning episodes, the instances are selected based on their performance on the task at hand, i.e., the fitness. Before continuing the learning, Gaussian noise is added to the meta-parameters with a predefined probability. We validate the OMPAC method by improving the state-of-the-art results in stochastic SZ-Tetris and in 10x10 Tetris by 31% and 84%, respectively, and by improving the learning speed and performance for deep Sarsa(λ) agents in the Atari 2600 domain.
Subfields: Artificial Intelligence, Computational Theory and Mathematics
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Hado van Hasselt
- David Silver
- Zhongwen Xu
- Sergey Levine
- Matteo Hessel
- Pieter Abbeel
- Shimon Whiteson
- R´emi Munos
In Computational Theory and Mathematics
- Hado van Hasselt
- David Silver
- Zhongwen Xu
- Sergey Levine
- Shimon Whiteson
- Xue Bin Peng
- Pieter Abbeel
- Michiel van de Panne
Citations in the corpus
- Meta-Gradient Reinforcement Learning (2018, methodology): “…adapting learning rate through gradient-based methods [Sutton, 1992, Schraudolph, 1999, Maclaurin et al., 2015, Pedregosa, 2016, Franceschi et al., 2017], Bayesian optimisation methods [Snoek et al., 2012], or evolution based hyper-parameter tuning [Jaderberg et al., 2017a, Elfwing et al., 2017].”
- Online Meta-Learning (2019, neutral): “One exception is work that adapts hyperparameters online (Elfwing et al., 2017; Meier et al., 2017; Baydin et al., 2017).”