What is Intrinsic Motivation? A Typology of Computational Approaches

By Pierre-Yves Oudeyer, Frederic Kaplan. 2007

834 citations recorded by the source corpus.

Intrinsic motivation, centrally involved in spontaneous exploration and curiosity, is a crucial concept in developmental psychology. It has been argued to be a crucial mechanism for open-ended cognitive development in humans, and as such has gathered a growing interest from developmental roboticists in the recent years. The goal of this paper is threefold. First, it provides a synthesis of the different approaches of intrinsic motivation in psychology. Second, by interpreting these approaches in a computational reinforcement learning framework, we argue that they are not operational and even sometimes inconsistent. Third, we set the ground for a systematic operational study of intrinsic motivation by presenting a formal typology of possible computational approaches. This typology is partly based on existing computational models, but also presents new ways of conceptualizing intrinsic motivation. We argue that this kind of computational typology might be useful for opening new avenues for research both in psychology and developmental robotics.

Subfields: Cognitive Neuroscience, Developmental and Educational Psychology, Experimental and Cognitive Psychology, Reinforcement Learning, Cognitive Science & Psychology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Developmental and Educational Psychology

In Experimental and Cognitive Psychology

In Reinforcement Learning

In Cognitive Science & Psychology

Citations in the corpus

Curiosity-Driven Exploration by Self-Supervised Prediction (2017, background): “Curiosity-driven exploration is a well studied topic in the reinforcement learning literature and a good summary can be found in (Oudeyer & Kaplan, 2009; Oudeyer et al., 2007).”
Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation (2016, background): “[20] categorize intrinsic motivation approaches into knowledge based methods, competence or goal based methods and morphological methods.”
Information seeking, curiosity and attention: computational and neural mechanisms (2013, positive): “2011) and may be quantified using Bayesian inference as a difference between a prior and posterior world model (Itti and Baldi 2009) or as a high prediction error for high-confidence states (Oudeyer and Kaplan 2007).”
Noisy Networks for Exploration (2017, background): “Many such terms have been proposed, including learning progress [26], compression progress [30], variational information maximisation [14] and prediction gain [4].”
Disentangling the independently controllable factors of variation by interacting with the world (2018, positive): “nate idea is that these representations do not need to be learned explicitly, and that learning can be guided through internal mechanisms of reward, usually called intrinsic motivation [Barto et al., Oudeyer and Kaplan, 2009, Salge et al., 2013, Gregor et al., 2017]. We build on a previously studied”
Dynamics-Aware Unsupervised Discovery of Skills (2019, methodology): “Another line of work that is conceptually close to our method copes with intrinsic motivation [51, 52, 58] which is used to drive the agent’s exploration.”
Large-Scale Study of Curiosity-Driven Learning (2018, background): “An alternative to “shaping” an extrinsic reward is to supplement it with dense intrinsic rewards [26], that is, rewards that are generated by the agent itself.”
Go-Explore: a New Approach for Hard-Exploration Problems (2019, background): “Prior to Go-Explore, the typical approach to sparse reward problems has been intrinsic motivation (IM) [4, 9–11], which supplies the RL agent with intrinsic rewards (IRs) that encourage exploration (augmenting or replacing extrinsic reward that comes from the environment).”
Learning Awareness Models (2018, background): “Some representative works include Oudeyer & Kaplan (2008; 2009); Sequeira et al. (2011); Still & Precup (2012); Bellemare et al. (2016); Martius et al. (2013); Schmidhuber (2008); Mohamed & Rezende (2015); Haber et al. (2018b;a) Some of the ideas here, in particular the notion of choosing actions…”
Semantic Exploration from Language Abstractions and Pretrained Representations (2022, background): “Our work is part of this last class of methods, where the agent is given an intrinsic reward for visiting diverse states over time [35].”
Formal Theory of Creativity, Fun, and Intrinsic Motivation (1990–2010) (2010, methodology): “The authors of [52] originally intended to present 2a and 2b as examples of 2, not as sub-categories (P.”
Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning (2017, methodology): “…last two decades, a series of computational and robotic models of intrinsically motivated exploration and learning in infants have been developed (Oudeyer and Kaplan, 2007; Baldassarre and Mirolli, 2013; Bazhydai et al., 2020), opening new theoretical perspectives in neuroscience and psychology…”
A free energy principle for a particular physics (2019, background): “Information gain is sometimes referred to as epistemic value or intrinsic motivation in artificial intelligence and robotics (Friston et al., 2015b; Oudeyer and Kaplan, 2007; Schmidhuber, 2010).”
Intrinsically motivated action-outcome learning and goal-based action recall: a system-level bio-constrained computational model. (2013, background): “…use of KB-IM mechanisms to acquire competence is quite common in the computational literature, since the classic models on IM (Schmidhuber, 1991b,a; Oudeyer et al., 2007) (but there are important models that use CB-IM mechanisms to acquire competence, e.g., Singh S. and Chentanez, 2005; Hart and…”
VIME: Variational Information Maximizing Exploration (2016, background): “We refer the reader to [21, 39] for an extensive review on curiosity and intrinsic rewards.”
Teacher–Student Curriculum Learning (2017, methodology): “The basic intuition is that the Student should practice those tasks more for which it is making most progress [10], while also practicing tasks that are at risk of being forgotten.”
Intrinsic motivations and open-ended development in animals, humans, and robots: an overview (2014, background): “These themes and concepts have been reviewed more extensively in other works (e.g., see Barto et al., 2004; Oudeyer and Kaplan, 2007; Mirolli and Baldassarre, 2013; Barto, 2013), but they are brieﬂy reported here both to meet the needs of the reader new to the ﬁeld and to introduce the concepts and…”