An information-theoretic approach to curiosity-driven reinforcement learning

By Susanne Still, Doina Precup. 2012

222 citations recorded by the source corpus.

Subfields: Cognitive Neuroscience, Statistical and Nonlinear Physics, Reinforcement Learning, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Statistical and Nonlinear Physics

In Reinforcement Learning

In Statistical Learning Theory

Citations in the corpus

Taming the Noise in Reinforcement Learning via Soft Updates (2015, methodology): “A KL penalty similar to ours is used in [35], in settings with known reward and transition functions, to encourage “curiosity”.”
Unifying Count-Based Exploration and Intrinsic Motivation (2016, methodology): “ themes, these novelty signals might be prediction error (Singh et al., 2004; Stadie et al., 2015), value error (Simsek and Barto, 2006), learning progress (Schmidhuber, 1991), or mutual information (Still and Precup, 2012; Mohamed and Rezende, 2015). The idea also ﬁnds roots in continual learning (”
Curiosity-Driven Exploration by Self-Supervised Prediction (2017, background): “Many approaches measure information gain for exploration (Little & Sommer, 2014; Still & Precup, 2012; Storck et al., 1995).”
Exploration by Random Network Distillation (2018, methodology): “This has motivated work on exploration via quantiﬁcation of uncertainty (Still & Precup, 2012; Houthooft et al., 2016) or prediction improvement measures (Schmidhuber, 1991a; Oudeyer et al., 2007; Lopes et al., 2012; Achiam & Sastry, 2017).”
Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning (2015, background): “…problem that forms the basis of a large number of applications in machine learning, including rate distortion theory [4], information bottleneck methods [26], population coding [1], curiosity-driven exploration [24, 19], model selection [3], and intrinsically-motivated reinforcement learning [20].”
Large-Scale Study of Curiosity-Driven Learning (2018, background): “Intrinsic Motivation: A family of approaches to intrinsic motivation reward an agent based on prediction error [2, 27, 36, 42], prediction uncertainty [11, 44], or improvement [19, 34] of a forward dynamics model of the environment that gets trained along with the agent’s policy.”
Learning Awareness Models (2018, background): “Some representative works include Oudeyer & Kaplan (2008; 2009); Sequeira et al. (2011); Still & Precup (2012); Bellemare et al. (2016); Martius et al. (2013); Schmidhuber (2008); Mohamed & Rezende (2015); Haber et al. (2018b;a) Some of the ideas here, in particular the notion of choosing actions…”
G-Learning: Taming the Noise in Reinforcement Learning via Soft Updates (2015, methodology): “Other works have used related methods for reinforcement learning [32, 33, 34, 35, 36].”
Non-parametric Policy Search with Limited Information Loss (2017, background): “Firstly, from the point of view of maximizing the cumulative reward under constraints on the communication bandwidth or data processing capacity (Tishby and Polani, 2011), or, equivalently, minimizing the policy complexity under a constraint on the policy’s value (Still and Precup, 2012).”
Learning concurrent motor skills in versatile solution spaces (2012, background): “REPS is an information theoretic policy search approach which is based on the insight that the information loss between two policy steps should be bounded [19], [20], [8].”
Influence-Based Multi-Agent Exploration (2019, unknown): “Mutual information is a critical statistical quantity and has been used in single-agent intrinsically motivated exploration (Rubin et al., 2012; Still & Precup, 2012; Salge et al., 2014; Mohamed & Rezende, 2015; Hyoungseok Kim, 2019) and multi-agent intention sharing and hiding (Strouse et al.,…”
Information-Theoretic Bounded Rationality (2015, unknown): “…systems [see e.g. Mitter and Newton, 2005, Kappen, 2005, Todorov, 2006, 2009, Theodorou et al., 2010, Theodorou, 2015, Still, 2009, Still and Precup, 2012, van den Broek et al., 2010, Friston, 2010, Peters et al., 2010, Tishby and Polani, 2011, Kappen et al., 2012, Neumann and…”
Decomposition of Uncertainty for Active Learning and Reliable Reinforcement Learning in Stochastic Systems (2017, background): “Within reinforcement learning, Bayesian notions of model uncertainty have a long history [Schmidhuber, 1991b,a, Dearden et al., 1999, Still and Precup, 2012, Sun et al., 2011, Maddison et al., 2017, Mihatsch and Neuneier, 2002].”
Active Inference, Curiosity and Insight (2017, background): “This resolution of uncertainty is closely related to satisfying artificial curiosity (Schmidhuber, 1991; Still & Precup, 2012) and speaks to the value of information (Howard, 1966), particularly in the context of evincing information necessary to realize rewards or payoffs (see Meder & Nelson,…”
Active Inference: A Process Theory (2017, background): “This resolution of uncertainty is closely related to satisfying artiﬁcial curiosity (Schmidhuber, 1991; Still & Precup, 2012) and speaks to the value of information (Howard, 1966).”
Learning Causal State Representations of Partially Observable Environments (2019, methodology): “Causal states, and the related information-theoretic notion of complexity, minimality, and suﬃciency are used to derive task-agnostic policies with an intrinsic exploration-exploitation trade oﬀ in [34, 35].”