Learning and exploration in action-perception loops

By Daniel Y. Little, Friedrich T. Sommer. 2013

126 citations recorded by the source corpus.

Discovering the structure underlying observed data is a recurring problem in machine learning with important applications in neuroscience. It is also a primary function of the brain. When data can be actively collected in the context of a closed action-perception loop, behavior becomes a critical determinant of learning efficiency. Psychologists studying exploration and curiosity in humans and animals have long argued that learning itself is a primary motivator of behavior. However, the theoretical basis of learning-driven behavior is not well understood. Previous computational studies of behavior have largely focused on the control problem of maximizing acquisition of rewards and have treated learning the structure of data as a secondary objective. Here, we study exploration in the absence of external reward feedback. Instead, we take the quality of an agent's learned internal model to be the primary objective. In a simple probabilistic framework, we derive a Bayesian estimate for the amount of information about the environment an agent can expect to receive by taking an action, a measure we term the predicted information gain (PIG). We develop exploration strategies that approximately maximize PIG. One strategy based on value-iteration consistently learns faster than previously developed reward-free exploration strategies across a diverse range of environments. Psychologists believe the evolutionary advantage of learning-driven exploration lies in the generalized utility of an accurate internal model. Consistent with this hypothesis, we demonstrate that agents which learn more efficiently during exploration are later better able to accomplish a range of goal-directed tasks. We will conclude by discussing how our work elucidates the explorative behaviors of animals and humans, its relationship to other computational models of behavior, and its potential application to experimental design, such as in closed-loop neurophysiology studies.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Electrical and Electronic Engineering

Citations in the corpus

Curiosity-Driven Exploration by Self-Supervised Prediction (2017, background): “Many approaches measure information gain for exploration (Little & Sommer, 2014; Still & Precup, 2012; Storck et al., 1995).”
DORA The Explorer: Directed Outreaching Reinforcement Action-Selection (2018, methodology): “This challenge has been addressed in a model-based setting: The idea is to use at every step the current estimate of the parameters of the MDP in order to compute, using dynamic programming, the future exploration bonus (Little & Sommer, 2014).”
World Discovery Models (2019, unknown): “…gain (Achiam & Sastry, 2017; Schmidhuber, 1991a; Lopes et al., 2012), and ﬁnally information gain (Shyam et al., 2018; Itti & Baldi, 2009; Little & Sommer, 2013; Frank et al., 2014; Houthooft et al., 2016). iv the last component is an RL algorithm that ﬁnds an optimal policy with respect…”
Eligibility Traces and Plasticity on Behavioral Time Scales: Experimental Support of NeoHebbian Three-Factor Learning Rules (2018, background): “Theoreticians have studied these ideas in the context of curiosity (Schmidhuber, 2010), information gain during active exploration (Storck et al., 1995; Sun et al., 2011; Schmidhuber, 2006; Sun et al., 2011; Little and Sommer, 2013; Friston et al., 2016), and via formal definitions of surprise (Stor”
VIME: Variational Information Maximizing Exploration (2016, methodology): “Our proposed method of utilizing information gain can be traced back to [22], and has been further explored in [17, 33, 34].”
Curiosity-driven Exploration in Deep Reinforcement Learning via Bayesian Neural Networks (2016, methodology): “Our proposed method of utilizing information gain can be traced back to [22], and has been further explored in [17, 33, 34].”
Active inference and epistemic value (2015, background): “…updating, and action selection may allow one to study, in a principled manner, how perception supports learning and how learning can result in different internal representations (Little & Sommer, 2013; Lungarella & Sporns, 2006; Ognibene &Baldassarre, 2014; Verschure, Voegtlin, & Douglas, 2003).”