Planning to Be Surprised: Optimal Bayesian Exploration in Dynamic Environments

By Yi Sun, Faustino J. Gomez, J. Urgen Schmidhuber. 2011

174 citations recorded by the source corpus.

To maximize its success, an AGI typically needs to explore its initially unknown world. Is there an optimal way of doing so? Here we derive an affirmative answer for a broad class of environments.

Subfields: Artificial Intelligence, Computational Theory and Mathematics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

Citations in the corpus

Curiosity-Driven Exploration by Self-Supervised Prediction (2017, background): “Schmidhuber (1991; 2010) and Sun et al. (2011) use surprise and compression progress as intrinsic rewards.”
Continual Learning for Robotics (2019, methodology): “This idea is related to online change detection algorithms (Sarkar and Meeker, 1998; Moens and Zenon, 2018) or Bayesian surprise (Sun et al., 2011) in ML.”
PowerPlay: Training an Increasingly General Problem Solver by Continually Searching for the Simplest Still Unsolvable Problem (2011, background): “xp loration seem obvious. One should analyze theoretically and experimentally under which conditions the creation of self-generated tasks can accelerate the solution to externally generated tasks—see [30, 54, 37, 38, 19, 4, 28, 62] for previous simple experimental studies in this vein. 6.1 Self-Refe”
Graph networks as learnable physics engines for inference and control (2018, background): “…for learning system identiﬁcation explicitly (Yu et al., 2017; Peng et al., 2017), learning policies that are robust to hidden property variations (Rajeswaran et al., 2016), and learning exploration strategies in uncertain settings (Schmidhuber, 1991; Sun et al., 2011; Houthooft et al., 2016).”
#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning (2016, methodology): “Bayesian RL methods (Kolter & Ng, 2009; Guez et al., 2014; Sun et al., 2011; Ghavamzadeh et al., 2015), which keep track of a distribution over MDPs, are an alternative to optimism-based methods.”
DORA The Explorer: Directed Outreaching Reinforcement Action-Selection (2018, background): “Such interpretation of exploration can also be found in other works (Schmidhuber (1991); Sun et al. (2011); Houthooft et al. (2016)).”
Surprise-Based Intrinsic Motivation for Deep Reinforcement Learning (2017, positive): “Conceptually and mathematically, our work is closest to prior work on curiosity and surprise [8, 19, 23, 24], although these works focus mainly on small finite MDPs.”
Toward an Integration of Deep Learning and Neuroscience (2016, methodology): “Adaptive sampling methods are also known in reinforcement learning that can achieve optimal Bayesian exploration of Markov Decision Process environments (Sun et al., 2011; Guez et al., 2012).”
Planning to Explore via Self-Supervised World Models (2020, background): “Sun et al. (2011) derived a model-based reinforcement learning agent that can optimize the inﬁnite-horizon information gain and experimented with it in tabular settings.”
The free energy principle made simpler but not too simple (2022, unknown): “This underwrites Bayesian experimental design [134] and active learning in statistics [135], intrinsic motivation and artificial curiosity in machine learning and robotics [136, 137, 138, 139, 140].”
Learning Novel Policies For Tasks (2019, methodology): “Some research uses the approach of curiosity-driven exploration (Schmidhuber, 1991; Sun et al., 2011; Conti et al., 2018) and applies these techniques to reinforcement learning (Houthooft et al., 2016; Hester & Stone, 2017).”
The strategic student approach for life-long exploration and learning (2012, background): “For instance by learning how to learn learning strategies [14] or by searching for the simplest still unexplained problem [58].”
An Introduction to Deep Reinforcement Learning (2018, unknown): “Inspired by the Bayesian setting, directed exploration can be done via heuristics of exploration bonus (Kolter and Ng, 2009) or by maximizing Shannon information gains (e.g., Sun et al., 2011).”
Action and Perception as Divergence Minimization (2020, background): “…(Lindley et al., 1956; MacKay, 1992b) that encourages inputs that are expected to convey the most bits about the representation to maximally reduce uncertainty in the belief, Information gain can be estimated by planning (Sun et al., 2011) or from past environment interaction (Schmidhuber, 1991).”
Ready Policy One: World Building Through Active Learning (2020, background): “Such issues have been highlighted previously (Schmidhuber, 1991; Sun et al., 2011), and motivate the design of our algorithm.”
Self-Supervised Exploration via Disagreement (2019, background): “This prediction error can also be formulated as surprise (Achiam & Sastry, 2017; Schmidhuber, 1991a; Sun et al., 2011).”
Decomposition of Uncertainty for Active Learning and Reliable Reinforcement Learning in Stochastic Systems (2017, background): “Within reinforcement learning, Bayesian notions of model uncertainty have a long history [Schmidhuber, 1991b,a, Dearden et al., 1999, Still and Precup, 2012, Sun et al., 2011, Maddison et al., 2017, Mihatsch and Neuneier, 2002].”
VIME: Variational Information Maximizing Exploration (2016, methodology): “with p(θ|ξt, at) = p(θ|ξt) as actions do not influence beliefs about the environment [17].”
An Exploration of Embodied Visual Exploration (2020, unknown): “Reconstruction-based exploration approaches also relate to classical informationgain based exploration, where the goal is to explore in order to reduce the uncertainty in the agent’s model of the world [13,63,66].”