Model-Free Episodic Control

By Charles Blundell, Benigno Uria, Alexander Pritzel, Yazhe Li, Yazhe Li, Avraham Ruderman, Avraham Ruderman, Joel Z. Leibo, Jack W. Rae, Daan Wierstra and 1 more. 2016

289 citations recorded by the source corpus.

State of the art deep reinforcement learning algorithms take many millions of interactions to attain human-level performance. Humans, on the other hand, can very quickly exploit highly rewarding nuances of an environment upon first discovery. In the brain, such rapid learning is thought to depend on the hippocampus and its capacity for episodic memory. Here we investigate whether a simple model of hippocampal episodic control can learn to solve difficult sequential decision-making tasks. We demonstrate that it not only attains a highly rewarding strategy significantly faster than state-of-the-art deep reinforcement learning algorithms, but also achieves a higher overall reward on some of the more challenging domains.

Subfields: Cellular and Molecular Neuroscience, Cognitive Neuroscience, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cellular and Molecular Neuroscience

In Cognitive Neuroscience

In Reinforcement Learning

Citations in the corpus

Conditional Image Synthesis with Auxiliary Classifier GANs (2016, background): “Better models of natural images also improve performance in semi-supervised learning tasks (Kingma et al., 2014; Sprin-genberg, 2015; Odena, 2016; Salimans et al., 2016) and re-inforcement learning problems (Blundell et al., 2016).”
Rainbow: Combining Improvements in Deep Reinforcement Learning (2017, background): “Episodic control (Blundell et al. 2016) also focuses on data efﬁciency, and was shown to be very effective in some domains.”
Replay in Deep Learning: Current Approaches and Missing Biological Elements (2021, positive): “For example (Pritzel et al., 2017; Lengyel & Dayan, 2008; Blundell et al., 2016) take inspiration from the role the hippocampus plays in making decisions to develop agents that learn much faster than other approaches.”
Scalable Recollections for Continual Lifelong Learning (2017, methodology): “Our method is complementary to recent work leveraging episodic storage to stabilize learning (Mnih et al. 2015; Blundell et al. 2016; Pritzel et al. 2017; Rebuffi, Kolesnikov, and Lampert 2017; Lopez-Paz and Ranzato 2017).”
Learning to Act by Predicting the Future (2016, background): “In another recent report, Blundell et al. (2016) considered a nonparametric approach to control and conducted experiments in a three-dimensional labyrinth.”
DARLA: Improving Zero-Shot Transfer in Reinforcement Learning (2017, methodology): “We used various RL algorithms (DQN, A3C and Episodic Control: Mnih et al., 2015; 2016; Blundell et al., 2016) to learn the source policy πS during stage two of the pipeline using the latent states sz acquired by β-VAE based models during stage one of the DARLA pipeline.”
Memory-based Parameter Adaptation (2018, background): “…et al., 2016; Santoro et al., 2016; Kaiser et al., 2017), language modelling (Merity et al., 2016; Grave et al., 2016), reinforcement learning (Blundell et al., 2016; Pritzel et al., 2017), machine translation (Bahdanau et al., 2014), and question answering (Weston et al., 2014), to name a…”
Cognitive Mapping and Planning for Visual Navigation (2017, background): “Others, such as [9, 50, 56], focus on the planning problem associated with navigation under full state information [56], designing strategies for faster learning via episodic control [9], or incorporate memory into DRL algorithms to ease generalization to new environments.”
Stabilizing Transformers for Reinforcement Learning (2019, background): “An alternative non-parametric perspective to memory stores an entire replay buffer of the agent’s past observations, which is made available for the agent to itself reason over either through ﬁxed rules (Blundell et al., 2016) or an attention operation (Pritzel et al., 2017).”
Semi-parametric Topological Memory for Navigation (2018, background): “, 2017) or episodic memory (Blundell et al., 2016; Pritzel et al., 2017) can potentially store information about the environment.”
Trial without Error: Towards Safe Reinforcement Learning via Human Intervention (2017, background): “This suggests a role for agents who initially explore systematically [24] and aggressively [6] and so encounter many catastrophes early on.”
Neural Episodic Control (2017, methodology): “Our work is in part inspired by the hypothesised role of the Hippocampus in decision making (Lengyel & Dayan, 2007; Blundell et al., 2016) and also by recent work on one-shot learning (Vinyals et al., 2016) and learning to remember rare events with neural networks (Kaiser et al., 2016).”
Training Agents using Upside-Down Reinforcement Learning (2019, background): “Episodic Control [Lengyel and Dayan, 2008; Blundell et al., 2016] is philosophically similar to our work in taking an alternate perspective on learning to act by directly associating good actions to high returns.”
Learning to Play in a Day: Faster Deep Reinforcement Learning by Optimality Tightening (2016, methodology): “The model-free episodic control approach evaluates state-action pairs based on episodic memory using k-nearest neighbors with hashing functions (Blundell et al., 2016).”
Toward an Integration of Deep Learning and Neuroscience (2016, background): “Alternatively, the brain could rapidly store an episodic memory and then retrieve it later without the need to perform slow gradient updates, which has proven to be useful for fast reinforcement learning in scenarios with limited available data (Blundell et al., 2016).”
Generalization of Reinforcement Learners with Working and Episodic Memory (2019, background): “Episodic memory has been shown to help reinforcement learning agents adapt more quickly and thereby boost data efﬁciency (Blundell et al., 2016; Pritzel et al., 2017; Hansen et al., 2018).”