VIME: Variational Information Maximizing Exploration

By Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, Pieter Abbeel. 2016

846 citations recorded by the source corpus.

Subfields: Reinforcement Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Trust-PCL: An Off-Policy Trust Region Method for Continuous Control (2017, background): “Indeed, other works which focus on exploration in continuous control have found the need to propose exploration-advanageous variants of these standard benchmarks (Haarnoja et al., 2017; Houthooft et al., 2016).”
Neural Approaches to Conversational AI (2018, neutral): “, 2017), exploration when parametric models like neural networks are used is an active research topic (Bellemare et al., 2016; Osband et al., 2016; Houthooft et al., 2016; Jiang et al., 2017).”
Uncertainty in Multitask Transfer Learning (2018, background): “This is indeed the case for Bayesian optimization [30], Bayesian active learning [12], continual learning [20], safe reinforcement learning [4], exploration-exploitation trade-off in reinforcement learning [17].”
Bayesian Model-Agnostic Meta-Learning (2018, background): “It brings us not only robustness to overﬁtting but also numerous beneﬁts such as improved prediction accuracy by Bayesian ensembling (Balan et al., 2015), active learning (Gal et al., 2016), and principled/safe exploration in reinforcement learning (Houthooft et al., 2016).”
Curiosity-Driven Exploration by Self-Supervised Prediction (2017, methodology): “Despite the sample efficiency of TRPO, we see that our ICM agents work significantly better than TRPO and TRPO-VIME, both in terms of convergence rate and accuracy.”
Automated Curriculum Learning for Neural Networks (2017, methodology): “Relationship to VIME.”
A Survey of Embodied AI: From Simulators to Research Tasks (2021, methodology): “The prediction error is used as the reward signal for reinforcement learning [63, 64].”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “This is generally achieved by defining a heuristic based on some measurement of surprise, information gain, compression, or empowerment (Schmidhuber, 1991; Klyubin et al., 2005; Kaplan and Oudeyer, 2006; Schmidhuber, 2008; Frank et al., 2014; Mohamed and Rezende, 2015; Bellemare et al., 2016; Houtho”
Exploration by Random Network Distillation (2018, methodology): “This has motivated work on exploration via quantiﬁcation of uncertainty (Still & Precup, 2012; Houthooft et al., 2016) or prediction improvement measures (Schmidhuber, 1991a; Oudeyer et al., 2007; Lopes et al., 2012; Achiam & Sastry, 2017).”
Benchmarking Model-Based Reinforcement Learning (2019, background): “While exploration and off-policy learning have been studied in [2, 10, 51, 23, 42, 18], it has been barely addressed on current model-based approaches.”
Emergent Tool Use From Multi-Agent Autocurricula (2019, background): “[45] Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel.”
MAVEN: Multi-Agent Variational Exploration (2019, background): “In a similar vein [21, 38] use auxiliary rewards to modify the RL objective towards a better tradeoff between exploration and exploitation.”
Diversity is All You Need: Learning Skills without a Reward Function (2018, background): “We also compared to VIME (Houthooft et al., 2016).”
Hindsight Experience Replay (2017, methodology): “Notice that using techniques for improving exploration (e.g. VIME (Houthooft et al., 2016), count-based exploration (Ostrovski et al., 2017) or bootstrapped DQN (Osband et al., 2016)) does not help here because the real problem is not in lack of diversity of states being visited, rather it is…”
Count-Based Exploration with Neural Density Models (2017, methodology): “Houthooft et al. (2016) used a second-order Taylor approximation of the prediction gain to drive exploration in continuous control.”
Noisy Networks for Exploration (2017, background): “Many such terms have been proposed, including learning progress [26], compression progress [30], variational information maximisation [14] and prediction gain [4].”
#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning (2016, neutral): “Figure 2 shows the results of TRPO (baseline), TRPO-SimHash, and VIME (Houthooft et al., 2016) on classic tasks MountainCar and CartPoleSwingup, the locomotion task HalfCheetah, and the hierarchical task SwimmerGather.”