First Experiments with PowerPlay

By Rupesh Kumar Srivastava, Bas R. Steunebrink, J. Urgen Schmidhuber. 2012

52 citations recorded by the source corpus.

Like a scientist or a playing child, POWERPLAY (Schmidhuber, 2011) not only learns new skills to solve given problems, but also invents new interesting problems by itself. By design, it continually comes up with the fastest to find, initially novel, but eventually solvable tasks. It also continually simplifies or compresses or speeds up solutions to previous tasks. Here we describe first experiments with POWERPLAY. A self-delimiting recurrent neural network SLIM RNN (Schmidhuber, 2012) is used as a general computational problem solving architecture. Its connection weights can encode arbitrary, self-delimiting, halting or non-halting programs affecting both environment (through effectors) and internal states encoding abstractions of event sequences. Our POWERPLAY-driven SLIM RNN learns to become an increasingly general solver of self-invented problems, continually adding new problem solving procedures to its growing skill repertoire. Extending a recent conference paper (Srivastava, Steunebrink, Stollenga, & Schmidhuber, 2012), we identify interesting, emerging, developmental stages of our open-ended system. We also show how it automatically self-modularizes, frequently re-using code for previously invented skills, always trying to invent novel tasks that can be quickly validated because they do not require too many weight changes affecting too many previous tasks.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Information Systems, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Information Systems

In Reinforcement Learning

Citations in the corpus

Adaptive Computation Time for Recurrent Neural Networks (2016, background): “A related architecture known as Self-Delimiting Neural Networks [26, 30] employs a halting neuron to end a particular update within a large, partially activated network; in this case however a simple activation threshold is used to make the decision, and no gradient with respect to halting time is p”
Recurrent World Models Facilitate Policy Evolution (2018, methodology): “M approach collapses C and M into a single network, and uses PowerPlay-like [82, 91] behavioural replay (where the behaviour of a teacher net is compressed into a student net [79]) to avoid forgetting old prediction and control skills when learning new ones.”
Solving Rubik's Cube with a Robot Hand (2019, methodology): “In similar spirit to our ADR, related works include PowerPlay [97, 103], POET [117] and different varieties of self-play [101, 102, 76].”
World Models (2018, methodology): “, 2017) and PowerPlay (Schmidhuber, 2013; Srivastava et al., 2012) also explores methods that lead to a natural curriculum learning (Schmidhuber, 2002), and we feel this is one of the more exciting research areas of reinforcement learning.”
CoMPS: Continual Meta Policy Search (2021, methodology): “Other recent methods support continual learning without ground truth task boundaries (Schmidhuber, 2011; Srivastava et al., 2012; Nguyen et al., 2017), but have not yet been demonstrated to perform well in reinforcement learning settings.”
Hindsight Experience Replay (2017, background): “Another interesting approach is PowerPlay (Schmidhuber, 2013; Srivastava et al., 2013) which is a general framework for automatic task selection.”
PowerPlay: Training an Increasingly General Problem Solver by Continually Searching for the Simplest Still Unsolvable Problem (2011, methodology): “epending on the user-deﬁned programming language, p i may invoke complex pre-wired subprograms (e.g., well-known learning algorithms) as primitive instructions—compare separate experimental an alysis [53, 52]. In general, p itself determines how much time to spend on SOLVER MODIFICATION—enough time ”
On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models (2015, methodology): “And POWERPLAY [243, 267] incrementally learns to become a more and more general algorithmic problem solver, by continually searching the space of possible pairs of new tasks and modifications of the current solver, until it finds a more powerful solver that, unlike the unmodified solver, solves all ”
One Big Net For Everything (2018, methodology): “First implementations of this simple principle were described in previous work on P OWER P LAY [75, 86].”
Training Agents using Upside-Down Reinforcement Learning (2019, background): “[Schmidhuber, 2013; Srivastava et al., 2013] uses “task inputs” to directly produce actions, uses SL on past trajectories to remember previous skills, and does not use value functions.”
Hindsight policy gradients (2017, background): “Recent examples of learning goal-conditional behavior include the works of Da Silva et al. [3], Schmidhuber [4], Srivastava et al. [5], Kupcsik et al. [6], Deisenroth et al. [7], Fabisch and Metzen [8], Schaul et al. [9], Zhu et al. [10], Held et al. [11].”
Progressive Reinforcement Learning with Distillation for Multi-Skilled Motion Control (2018, methodology): “The Powerplay method provides a general framework for training an increasingly general problem solver (Schmidhuber, 2011; Srivastava et al., 2012).”
Competitive Experience Replay (2019, background): “A general framework for automatic task selection is Powerplay (Schmidhuber, 2013; Srivastava et al., 2013), which proposes an asymptotically optimal way of selecting tasks from a set of tasks with program search, and use replay buffer to avoid catastrophic forgetting.”
Compete to Compute (2013, background): “Nonetheless, networks employing local competition have existed since the late 80s [21], and, along with [22], serve as a primary inspiration for the present work.”
Meta-learning curiosity algorithms (2020, background): “However, there are many exploration algorithm classes that we do not cover, such as those focusing on generating goals (Srivastava et al., 2013; Kulkarni et al., 2016; Florensa et al., 2018), learning progress (Oudeyer et al., 2007; Schmidhuber, 2008; Azar et al., 2019), generating diverse skills…”
Reinforcement Learning Upside Down: Don't Predict Rewards - Just Map Them to Actions (2019, methodology): “The P OWER P LAY framework [51, 62] also uses a similar approach, learning one task after another, using environment-independent replay of behavioral traces (or functionally equivalent but more efﬁcient approaches) to avoid forgetting previous skills and to compress or speed up previously found,…”