Neural Episodic Control

By Alexander Pritzel, Benigno Uria, Sriram Srinivasan, Adrià Puigdomènech Badia, Oriol Vinyals, Demis Hassabis, Daan Wierstra, Charles Blundell. 2017

405 citations recorded by the source corpus.

Deep reinforcement learning methods attain super-human performance in a wide range of environments. Such methods are grossly inefficient, often taking orders of magnitudes more data than humans to achieve reasonable performance. We propose Neural Episodic Control: a deep reinforcement learning agent that is able to rapidly assimilate new experiences and act upon them. Our agent uses a semi-tabular representation of the value function: a buffer of past experience containing slowly changing state representations and rapidly updated estimates of the value function. We show across a wide range of environments that our agent learns significantly faster than other state-of-the-art, general purpose deep reinforcement learning agents.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Reinforcement Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Reinforcement Learning

In Meta & Continual Learning

Citations in the corpus

Set Transformer (2018, background): “The differential neural dictionary (Pritzel et al., 2017) stores previous experience as key-value pairs and uses this to process queries.”
Self-Imitation Learning (2018, background): “, 2016) and NEC (Pritzel et al., 2017) scaled up this idea to complex domains.”
Replay in Deep Learning: Current Approaches and Missing Biological Elements (2021, neutral): “For example (Pritzel et al., 2017; Lengyel & Dayan, 2008; Blundell et al., 2016) take inspiration from the role the hippocampus plays in making decisions to develop agents that learn much faster than other approaches.”
Scalable Recollections for Continual Lifelong Learning (2017, methodology): “Our method is complementary to recent work leveraging episodic storage to stabilize learning (Mnih et al. 2015; Blundell et al. 2016; Pritzel et al. 2017; Rebuffi, Kolesnikov, and Lampert 2017; Lopez-Paz and Ranzato 2017).”
Memory-based Parameter Adaptation (2018, methodology): “…= f γ ( x ) in the memory M : C = { ( h k , w k , q ) are weightings of each of the retrieved neighbours according to their closeness to the query f γ ( x ) . kern ( h, q ) is a kernel function which, following (Pritzel et al., 2017), we choose as kern ( h, q ) = 1 + (cid:107) h − q (cid:107) 22 .”
A Brief Survey of Deep Reinforcement Learning (2017, neutral): “Alternatively, differentiable memories can be used as approximate hash tables, allowing DRL algorithms to store and retrieve successful experiences to facilitate rapid learning [105].”
Stabilizing Transformers for Reinforcement Learning (2019, background): “An alternative non-parametric perspective to memory stores an entire replay buffer of the agent’s past observations, which is made available for the agent to itself reason over either through ﬁxed rules (Blundell et al., 2016) or an attention operation (Pritzel et al., 2017).”
Semi-parametric Topological Memory for Navigation (2018, background): “Systems equipped with general-purpose LSTM memory (Mnih et al., 2016; Pathak et al., 2017; Jaderberg et al., 2017; Mirowski et al., 2017) or episodic memory (Blun-dell et al., 2016; Pritzel et al., 2017) can potentially store information about the environment.”
Training Agents using Upside-Down Reinforcement Learning (2019, methodology): “Ideas from Episodic Control were later combined with traditional value-based RL [Pritzel et al., 2017], which is also a promising future direction for R L .”
Search Engine Guided Non-Parametric Neural Machine Translation (2017, background): “A similar approach was very recently proposed for deep reinforcement learning by Pritzel et al. (2017), where they store pairs of observed state and the corresponding (estimated) value in a key-value memory to build a non-parametric deep Q network.”
Learning Attractor Dynamics for Generative Memory (2018, methodology): “Various forms of external memory have been used to augment neural networks [5, 14, 25, 29, 31, 32].”
Been There, Done That: Meta-Learning with Episodic Recall (2018, methodology): “First, the DND provides a mechanism for passing gradients through the retrieval process (Pritzel et al., 2017); future work should explore the possibilities of using this learning pathway for epL2RL.”
A Simple Cache Model for Image Recognition (2018, background): “Similar cache models have recently been proposed to improve the sample efficiency in reinforcement learning [13] and one-shot learning problems [8] as well.”
Towards Better Interpretability in Deep Q-Networks (2018, result): “Architecturally, our network is similar to the network first proposed by (Pritzel et al. 2017).”
Combinatorial optimization and reasoning with graph neural networks (2021, background): “This effectively endows PMP with an episodic memory (Pritzel et al., 2017) of its past computations.”
The Benchmark Lottery (2021, methodology): “Alternatively in some publication, methods are evaluated for each 2 × 10 5 frames [Pritzel et al., 2017], while in others methods are evaluated every 10 6 frames [Mnih et al., 2013, 2016].”
On the link between conscious function and general intelligence in humans and machines (2022, background): “Whilst the majority of deep RL algorithms use experience to update the policy and/or value functions offline, some RL methods have taken direct inspiration from episodic memory to create policies that are directly conditioned on past experiences (Blundell et al., 2016; Pritzel et al., 2017).”