Episodic Curiosity through Reachability

By Nikolay Savinov, Anton Raichuk, Raphaël Marinier, Damien Vincent, Marc Pollefeys, Timothy P. Lillicrap, Sylvain Gelly. 2018

291 citations recorded by the source corpus.

Rewards are sparse in the real world and most of today's reinforcement learning algorithms struggle with such sparsity. One solution to this problem is to allow the agent to create rewards for itself - thus making rewards dense and more suitable for learning. In particular, inspired by curious behaviour in animals, observing something novel could be rewarded with a bonus. Such bonus is summed up with the real task reward - making it possible for RL algorithms to learn from the combined reward. We propose a new curiosity method which uses episodic memory to form the novelty bonus. To determine the bonus, the current observation is compared with the observations in memory. Crucially, the comparison is done based on how many environment steps it takes to reach the current observation from those in memory - which incorporates rich information about environment dynamics. This allows us to overcome the known "couch-potato" issues of prior work - when the agent finds a way to instantly gratify itself by exploiting actions which lead to hardly predictable consequences. We test our approach in visually rich 3D environments in ViZDoom, DMLab and MuJoCo. In navigational tasks from ViZDoom and DMLab, our agent outperforms the state-of-the-art curiosity method ICM. In MuJoCo, an ant equipped with our curiosity module learns locomotion out of the first-person-view curiosity only.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

A Survey of Embodied AI: From Simulators to Research Tasks (2021, background): “sidered, such as area [45, 73] and interesting objects [56, 74].”
Large-Scale Long-Tailed Recognition in an Open World (2019, background): “γ measures the reachability [43] of an input’s direct feature v to the memory M — the minimum distance between the direct feature and the discriminative centroids:”
Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions (2019, background): “[44] extend the archive-based mechanism from NS that determines novelty in behaviors to determine novelty in the observation space and then to formulate the novelty bonus as an auxiliary reward.”
Search on the Replay Buffer: Bridging Planning and Reinforcement Learning (2019, methodology): “On this task, we evaluate four state-of-the-art prior methods: hindsight experience replay (HER) (Andrychowicz et al., 2017), distributional RL (C51) (Bellemare et al., 2017), semi-parametric topological memory (SPTM) (Savinov et al., 2018a), and value iteration networks (VIN) (Tamar et al., 2016).”
Exploration in Deep Reinforcement Learning: A Survey (2022, background): “Prediction error methods have also shown the ability to solve the couch-potato (noisy-TV) problem by storing observations in a memory buffer [60].”
Never Give Up: Learning Directed Exploration Strategies (2020, methodology): “Our work differs from Savinov et al. (2018) in that it is not specialised to navigation tasks, our method incorporates a long-term intrinsic reward and is able to separate exploration and exploitation policies.”
Continuous control for robot based on deep reinforcement learning (2019, methodology): “Google Brain improved the curiosity method by introducing a episodic memory to determine the intrinsic reward [30].”
Keeping Your Distance: Solving Sparse Reward Tasks Using Self-Balancing Shaped Rewards (2019, result): “We compare the performance of Sibling Rivalry to a Grid Oracle baseline (Savinov et al., 2019).”
Learning Affordance Landscapes for Interaction Exploration in 3D Environments (2020, background): “Recent work formulates intrinsic motivation based on curiosity [43, 9, 27], novelty [51, 7], and empowerment [37] to improve video game playing agents (e.”
Self-Attentional Credit Assignment for Transfer in Reinforcement Learning (2019, unknown): “We use PPO [Schulman et al. , 2017] for in-domain experiments in DMLab, with identical hyper-parameters as in Episodic Curiosity [Savinov et al. , 2019], whose code is open-source.”
Shaping embodied agent behavior with activity-context priors from egocentric video (2021, background): “Exploration for navigation and interaction Exploration strategies for visual navigation encourage efﬁcient policy learning by rewarding agents for covering area [12, 10, 17], visiting new states [50, 4], expanding the frontier [46], anticipating maps [49], or via intrinsic motivation [41, 6].”
Action and Perception as Divergence Minimization (2020, methodology): “…is often estimated by learning a density model of individual inputs as in pseudo-counts (Bellemare et al., 2016), latent variable models as in SkewFit (Pong et al., 2019), unnormalized models as in RND (Burda et al., 2018), and non-parameteric models as in reachability (Savinov et al., 2018).”
An Introduction to Deep Reinforcement Learning (2018, unknown): “The agent can thus seek to take actions that adversarially challenge its knowledge of the environment (Savinov et al., 2018).”
Object Goal Navigation using Goal-Oriented Semantic Exploration (2020, methodology): “These include methods which use recurrent neural networks [30, 25, 6, 37, 21, 7, 38, 42], structured spatial representations [16, 33, 8, 19, 15] and topological representations [35, 36].”
Agent57: Outperforming the Atari Human Benchmark (2020, background): “…et al., 2018; Eysenbach et al., 2018) and intrinsic motivation (Schmidhuber, 1991; Oudeyer et al., 2007; Barto, 2013; Bellemare et al., 2016; Ostrovski et al., 2017; Fu et al., 2017; Tang et al., 2017; Burda et al., 2018; Choi et al., 2018; Savinov et al., 2018; Puigdomènech Badia et al., 2020).”
Bayesian Relational Memory for Semantic Visual Navigation (2019, background): “Savinov et al. [47] and Kuang et al. [18] also consider extracting visual concepts dynamically by treating every received input frame as an individual concept and storing them in the memory.”
An Exploration of Embodied Visual Exploration (2020, unknown): “First, prior work evaluates on different simulation environments such as SUN360 [42, 49], ModelNet [42, 49], VizDoom [39, 45], SUNCG [14], DeepMindLab [45], and Matterport3D [13].”
Neural Topological SLAM for Visual Navigation (2020, background): “This includes spatial memories [17, 30] and topological approaches [8, 13, 34, 35, 42, 45].”
Learning to Explore using Active Neural SLAM (2020, background): “Recent works from Chen et al. (2019); Savinov et al. (2019); Fang et al. (2019) attack this problem via learning.”
POET: open-ended coevolution of environments and their optimized solutions (2019, background): “[48], adding NS to ES [8], and the recent success”