Reinforcement Learning with Self-Modifying Policies

By J. Urgen Schmidhuber, Jieyu Zhao, N. Schraudolph. 1998

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Deep learning in neural networks: An overview (2014, background): “omputational effort.) 6.8 Universal RL General purpose learning algorithms may improve themselves in open-ended fashion and environmentspeciﬁc ways in a lifelong learning context (Schmidhuber, 1 987; Schmidhuber et al., 1997b,a; Schaul and Schmidhuber, 2010). The most general type of RL is constrain”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “Self-modifying policies (Schmidhuber, Zhao, and Schraudolph, 1998) provide a general framework for meta-learning in continuing RL environments.”
On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models (2015, methodology): “General methods for incremental multitask RL and algorithmic transfer learning that are not NN-specific include the evolutionary ADATE system [182], the Success-Story Algorithm for SelfModifying Policies running on general-purpose computers [233, 252, 251], and the Optimal Ordered Problem Solver [23”
Optimal Ordered Problem Solver (2002, methodology): “e introduced the concept of incremental search for improved, probabilistically generated code that modiﬁes the probability distribution on the possible code continuations: incremental self-improvers (Schmidhuber et al., 1997a) use the success-story algorithm SSA to undo those self-generated probabil”
Multimodal Model-Agnostic Meta-Learning via Task-Aware Modulation (2019, background): “Along with few-shot classification and regression, reinforcement learning (RL) has been a central problem where meta-learning has been studied [40, 39, 52, 5, 25, 35].”
Gödel Machines: Fully Self-referential Optimal Universal Self-improvers (2007, methodology): “If the learning algorithm has modifiable components represented as part of the policy, then we speak of a self-modifying policy (SMP) [48].”
Artificial curiosity based on discovering novel algorithmic predictability through coevolution (1999, methodology): “For such reasons I use Incremental Self-Improvement (IS) [21, 20] (see details in [17]) to deal with both modules' complex spatio-temporal credit assignment problem.”
Ranking Learning Algorithms: Using IBL and Meta-Learning on Accuracy and Time Results (2003, background): “Another different approach to meta-learning is based on so-called self-modifying policies ( Schmidhuber, Zhao, & Schraudolph, 1997 ).”
Driven by Compression Progress: A Simple Principle Explains Essential Aspects of Subjective Beauty, Novelty, Surprise, Interestingness, Attention, Curiosity, Creativity, Art, Science, Music, Jokes (2008, methodology): “…the computational power of controller and predictor by implementing them as co-evolving, symmetric, opposing modules consisting of self-modifying probabilistic programs [97, 98] written in a universal programming language [18, 111] allowing for loops, recursion, and hierarchical structures.”
The Tools Challenge: Rapid Trial-and-Error Learning in Physical Problem Solving (2019, unknown): “…as an exciting direction for tackling multi-task reinforcement learning in model-based and model-free settings (Finn, Abbeel, & Levine, 2017; Rakelly, Zhou, Finn, Levine, & Quillen, 2019; Gupta, Mendonca, Liu, Abbeel, & Levine, 2018; Schmidhuber, Zhao, & Schraudolph, 1998; Wang et al., 2018) .”
Goedel Machines: Self-Referential Universal Problem Solvers Making Provably Optimal Self-Improvements (2003, methodology): “Solely by self-modiﬁcations our SMP/SSA-based learners solved a complex task in a partially observable environment whose state space is far bigger than most found in the literature [53].”
Optimal Artiﬁcial Curiosity, Creativity, Music, and the Fine Arts (2005, methodology): “Both modules tried to maximize their money using a rather general RL algorithm designed for complex stochastic policies [44,45].”