A possibility for implementing curiosity and boredom in model-building neural controllers

By J. Urgen Schmidhuber. 1991

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Learning Parameterized Skills (2012, background): “, [60, 61]), where he posits that intrinsically motivated agents are interested in learnable but yet unknown regularities, but bored by predictable or inherently unpredictable events.”
Recurrent World Models Facilitate Policy Evolution (2018, methodology): “Many basic concepts ﬁrst explored in the 1980s for feed-forward neural networks (FNNs) [56, 58, 72, 104, 105] and in the 1990s for RNNs [74, 75, 76, 78] laid some of the groundwork for Learning to Think [83].”
Unifying Count-Based Exploration and Intrinsic Motivation (2016, background): “Closely related to prediction gain is Schmidhuber (1991)’s notion of compression progress, which equates novelty with an agent’s improvement in its ability to compress its past.”
World Models (2018, methodology): “Large RNNs are highly expressive models that can learn rich spatial and temporal representations of data.”
Curiosity-Driven Exploration by Self-Supervised Prediction (2017, background): “Measuring learning progress instead of prediction error has been proposed in the past as one solution (Schmidhuber, 1991).”
Automated Curriculum Learning for Neural Networks (2017, methodology): “…2013), where various indicators of learning progress have been used as reward signals to encourage exploration, including compression progress (Schmidhuber, 1991), information acquisition (Storck et al., 1995), Bayesian surprise (Itti and Baldi, 2009), prediction gain (Bellemare et al., 2016)…”
Active learning of inverse models with intrinsically motivated goal exploration in robots (2013, positive): “…havebeenproposedindevelopmentalrobotics,suchasthosebased on the psychological concept of intrinsic motivations [54,46,55] which relate to mechanisms that drive a learning agent to perform different activities for their own sake, without requiring any exter-nal reward [39,33,56–61,41 ,62 –65 ].”
Exploration by Random Network Distillation (2018, methodology): “Another class of exploration methods rely on errors in predicting dynamics (Schmidhuber, 1991b; Stadie et al., 2015; Achiam & Sastry, 2017; Pathak et al., 2017; Burda et al., 2018).”
Regularized Behavior Value Estimation (2021, background): “In the online case, this over-estimation incen-tivizes agents to explore actions of high expected reward, thus learning by trial-and-error (Schmidhuber, 1991).”
Emergent Tool Use From Multi-Agent Autocurricula (2019, background): “Another paradigm are transition-based methods, in which agents are rewarded for high prediction error in a learned forward or inverse dynamics model [42, 43, 44, 45, 46, 47, 48, 49].”
Troubling Trends in Machine Learning Scholarship (2018, background): “n connotations without the need to argue for them. This often manifests in anthropomorphic characterizations of tasks (reading comprehension [31] and music composition [59]) and techniques (curiosity [66] and fear [48]). A number of papers name components of proposed models in a manner suggestive of”
Learning Latent Plans from Play (2019, background): “But how do we define and implement robotic play, with all the same crucial properties of play previously identified? Voluntary and varied object interaction could in principle be collected by any agent equipped with 1) curiosity, boredom, or some intrinsic motivation drive [16, 51, 53] and 2) a foun”
On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models (2015, methodology): “M has m+n+o input units to receive all(t) at time t < tdeath, and m + n output units to produce a prediction pred(t + 1) ∈ R of sense(t + 1) [223, 226, 222, 227].”
A Brief Survey of Deep Reinforcement Learning (2017, neutral): “UCB can also be considered one way of implementing intrinsic motivation, which is a general concept that advocates decreasing uncertainty/making progress in learning about the environment [101].”
Large-Scale Study of Curiosity-Driven Learning (2018, background): “Examples of intrinsic reward include “curiosity” [11, 22, 27, 35, 40] which uses prediction error as reward signal, and “visitation counts” [3, 20, 24, 30] which discourages the agent from revisiting the same states.”
One Big Net For Everything (2018, methodology): “For this purpose, ONE has m + n special output units to produce for t t T a prediction pred ( t ) ∈ R m + n of sense ( t + 1) [55, 56, 59, 54, 60] from ONE’s previous observations and actions, which are in principle accessible to ONE through (recurrent) connections.”
Zero-Shot Visual Imitation (2018, methodology): “Furthermore, using navigation experiments in VizDoom environment, we show that (GSP) learned using curiosity-driven exploration (Oudeyer et al., 2007; Pathak et al., 2017; Schmidhuber, 1991) can more accurately follow demonstrations as compared to using random exploration data for learning the GSP.”