Exploration in Model-based Reinforcement Learning by Empirically Estimating Learning Progress

By Manuel Lopes, Tobias Lang, Marc Toussaint, Pierre-Yves Oudeyer. 2012

209 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Curiosity-Driven Exploration by Self-Supervised Prediction (2017, background): “However, estimating learnability is a non-trivial problem (Lopes et al., 2012).”
Exploration by Random Network Distillation (2018, methodology): “To avoid the undesirable factors 2 and 3, methods such as those by Schmidhuber (1991a); Oudeyer et al. (2007); Lopes et al. (2012); Achiam & Sastry (2017) instead use a measurement of how much the prediction model improves upon seeing a new datapoint.”
Variational Bayesian Reinforcement Learning with Regret Bounds (2018, background): “ respect to a set of axioms [107]. In the reinforcement literature there is a lot of work on exploration strategies such as optimism, curiosity, intrinsic motivation, surprise, novelty seeking, etc., [80, 33, 43, 49, 56, 21]. Assuming these agents are rational, then in the language of utility theory”
Count-Based Exploration with Neural Density Models (2017, methodology): “Quantities related to prediction gain have been used for similar purposes in the intrinsic motivation literature (Lopes et al., 2012), where they measure an agent’s learning progress (Oudeyer et al., 2007).”
Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play (2017, methodology): “These techniques can be applied in encouraging exploration in the context of reinforcement learning, for example (Bellemare et al., 2016; Strehl & Littman, 2008; Lopes et al., 2012; Tang et al., 2016); Roughly, these use some notion of the novelty of a state to give a reward.”
Large-Scale Study of Curiosity-Driven Learning (2018, background): “Intrinsic Motivation: A family of approaches to intrinsic motivation reward an agent based on prediction error [2, 27, 36, 42], prediction uncertainty [11, 44], or improvement [19, 34] of a forward dynamics model of the environment that gets trained along with the agent’s policy.”
Surprise-Based Intrinsic Motivation for Deep Reinforcement Learning (2017, background): “[11] investigated exploration driven by learning progress and proved theoretical guarantees for their approach in the finite MDP case, but they did not address the question of scaling their approach to continuous or high-dimensional MDPs.”
Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models (2015, methodology): “, as the size of state spaces increases, these methods quickly become intractable. A number of prior methods also examine various techniques for using models and prediction to incentivize exploration [12, 13, 14, 15]. However, such methods typically operate directly on the transition matrix of a dis”
Active choice of teachers, learning strategies and goals for a socially guided intrinsic motivation learner (2012, background): “· intrinsic motivation in psychology [38] which triggers spontaneous exploration and curiosity in humans, which recently led to novel robotic and machine active learning methods which outperform traditional active learning methods [6, 24]”
Unsupervised Curricula for Visual Meta-Reinforcement Learning (2019, background): “Some prior works consider unsupervised pre-training for efficient RL, but these works typically focus on settings in which exploration is not as much of a challenge [63, 17, 14], focus on goal-conditioned policies [44, 40], or have not been shown to scale to high-dimensional visual observation space”
Transfer Reinforcement Learning with Shared Dynamics (2017, background): “(Lopes et al. 2012) and (Osband, Roy, and Wen 2016) are also proposing interesting alternative options for guiding the exploration.”
Unifying Count-Based Exploration and Intrinsic Motivation (2016, background): “More recently, Lopes et al. (2012) showed the relationship between time-averaged prediction gain and visit counts in a tabular setting; their result is a special case of Theorem 2.”
SMiRL: Surprise Minimizing Reinforcement Learning in Unstable Environments (2021, methodology): “Learning progress methods that minimize model parameter entropy (Lopes et al., 2012; Kim et al., 2020) avoid the issues novelty-based methods have with noisy distractors.”
Automated Curriculum Learning by Rewarding Temporally Rare Events (2018, background): “One way of implementing intrinsic motivation is to model the expected learning progress ζ(s, a) of a state-action pair [22].”
Multi-Armed Bandits for Intelligent Tutoring Systems (2013, background): “In this context, they were shown to have high robustness and performances, especially when learning progress is estimated empirically and incrementally [38, 7], i.”
Curiosity-driven Exploration in Deep Reinforcement Learning via Bayesian Neural Networks (2016, background): “Other metrics for curiosity have also been proposed, including prediction error [10, 35], prediction error improvement [36], and leverage [14].”
An exploration strategy for non-stationary opponents (2017, unknown): “A recent approach based on R-max that takes into account possible changes in the dynamics of the environment is ζ -R-max [34], which does not assume that the visitation counts translate directly to model certainty.”