Prioritized memory access explains planning and hippocampal replay

By Marcelo Gomes Mattar, Nathaniel D. Daw. 2017

346 citations recorded by the source corpus.

To make decisions, animals must evaluate candidate choices by accessing memories of relevant experiences. Yet little is known about which experiences are considered or ignored during deliberation, which ultimately governs choice. We propose a normative theory predicting which memories should be accessed at each moment to optimize future decisions. Using nonlocal ‘replay’ of spatial locations in hippocampus as a window into memory access, we simulate a spatial navigation task in which an agent accesses memories of locations sequentially, ordered by utility: how much extra reward would be earned due to better choices. This prioritization balances two desiderata: the need to evaluate imminent choices versus the gain from propagating newly encountered information to preceding locations. Our theory offers a simple explanation for numerous findings about place cells; unifies seemingly disparate proposed functions of replay including planning, learning, and consolidation; and posits a mechanism whose dysfunction may underlie pathologies like rumination and craving. Mattar and Daw propose a normative theory predicting which memories should be accessed at each moment to optimize future decisions. This theory offers a simple explanation for numerous findings about hippocampal replay, bridging planning and learning.

Subfields: Cellular and Molecular Neuroscience, Cognitive Neuroscience

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cellular and Molecular Neuroscience

In Cognitive Neuroscience

Citations in the corpus

Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “Additionally, there is strong evidence in the literature of hippocampal replay facilitating the model-based planning process for human RL (Mattar and Daw, 2018; Momennejad, Otto, Daw, and Norman, 2018; Vikbladh, Meager, King, Blackmon, Devinsky, Shohamy, Burgess, and Daw, 2019; Momennejad, 2020).”
Cognitive Architectures for Language Agents (2023, background): “Biological agents, however, do not have this luxury: they must balance learning against external actions in their lifetime, choosing when and what to learn (Mattar and Daw, 2018).”
Presynaptic stochasticity improves energy efficiency and helps alleviate the stability-plasticity dilemma (2021, background): “As a growing body of literature suggests, artificial neural networks (ANNs) match several aspects of biological neuronal networks in various goal-driven situations (Kriegeskorte, 2015; Yamins and DiCarlo, 2016; Kell et al., 2018; Banino et al., 2018; Cueva and Wei, 2018; Mattar and Daw, 2018).”
Automatic Recall Machines: Internal Replay, Continual Learning and the Brain (2020, background): “This forms a parallel with the biological brain, where increased hippocampal replay leads to increased task performance or reward [Schuck and Niv, 2019] and vice versa [Mattar and Daw, 2018].”
DREAM Architecture: a Developmental Approach to Open-Ended Learning in Robotics (2020, methodology): “nomenon. One recent hypothesis is that such a replay might be useful to bootstrap reinforcement learning by using an internal model of the task structure to update state-action value functions ofﬂine [174,59], similarly to the Dyna architecture [175]. Nevertheless, applying this type of neuro-inspir”
Analogues of mental simulation and imagination in deep learning (2019, neutral): “(a) Dyna [59] is a background planning method which uses simulated experience in place of real experience.”
Learning Structures: Predictive Representations, Replay, and Generalization (2020, background): “A recent model of prioritized forward and reverse replay leverages these properties and consults SR to compare the value of taking different mental actions during planning in order to determine what replay content maximizes planning outcomes [56].”
Hippocampal Contributions to Model-Based Planning and Spatial Memory (2019, neutral): “In spatial tasks, sequential activations of place-selective cells are hypothesized to provide, not only a mnemonic function through supporting reactivation of previously traversed trajectories, but a planning function by generating novel place cell sequences, based on the learned contingencies betwe”