The hippocampus as a predictive map

By Kimberly L. Stachenfeld, Matthew Botvinick, Samuel J. Gershman. 2017

927 citations recorded by the source corpus.

A cognitive map has long been the dominant metaphor for hippocampal function, embracing the idea that place cells encode a geometric representation of space. However, evidence for predictive coding, reward sensitivity and policy dependence in place cells suggests that the representation is not purely spatial. We approach this puzzle from a reinforcement learning perspective: what kind of spatial representation is most useful for maximizing future reward? We show that the answer takes the form of a predictive representation. This representation captures many aspects of place cell responses that fall outside the traditional view of a cognitive map. Furthermore, we argue that entorhinal grid cells encode a low-dimensionality basis set for the predictive representation, useful for suppressing noise in predictions and extracting multiscale structure for hierarchical planning.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, neutral): “…and Mapping (SLAM) (Thrun et al., 2005) procedures, either partially or fully and embedded internally or externally, into DRL network architectures, with the intention to cultivate more sophisticated navigation capabilities in DRL agents (Stachenfeld et al., 2017; Kanitscheider and Fiete, 2017).”
Learning Structures: Predictive Representations, Replay, and Generalization (2020, methodology): “As such, the eigenvectors of the policydependent SR may lead to grid fields that over-represent often-visited states, tiling “more space” near highly visited locations [15].”
What Is a Cognitive Map? Organizing Knowledge for Flexible Behavior. (2018, background): “496 Neuron 100, October 24, 2018 step simulation (Baram et al., 2018; Stachenfeld et al., 2017).”
Deep learning in business analytics and operations research: Models, applications and managerial implications (2018, unknown): “Hence, the idea of activation functions is biologically inspired in the sense that they re-semble neurons in the human brain, which also have to receive a certain stimuli in order to be activated ( Stachenfeld, Botvinick, & Gershman, 2017 ).”
Learning One Representation to Optimize All Rewards (2021, background): “These loosely correspond to long-range (low-frequency) behavior [MM07]: presumably, 𝐹 and 𝐵 will learn spatially smooth rewards ﬁrst.”
Generative Predictive Codes by Multiplexed Hippocampal Neuronal Tuplets. (2018, unknown): “Intrinsic-Unlikely Signals Selectively Update the Generative Internal Model During spatial exploration, incorporation of novel information from the external world would result in perturbation and editing of intrinsically generated neuronal sequences and encoding of the difference from the internal m”
Prioritized memory access explains planning and hippocampal replay (2017, unknown): “Although the need term is straightforward (it is the SR [41], which the brain has been proposed to track for other reasons [49, 42]), the calculation of gain, as we define it, requires that the agent knows the effect of a backup on its policy prior to deciding whether to perform it.”
Eigenoption Discovery through the Deep Successor Representation (2017, methodology): “Moreover, the eigenvectors generated from its eigendecomposition are equivalent to proto-value functions (Stachenfeld et al., 2014; 2017) and to slow feature analysis (Sprekeler, 2011).”
A dedicated population for reward coding in the hippocampus (2018, unknown): “In the framework of reinforcement learning, reward-predictive cells are consistent withmodels in which the role of the hippocampus is to support prediction of expected reward, such as the successor representation theory (Dayan, 1993; Stachenfeld et al., 2017).”
Spectral Inference Networks: Unifying Spectral Methods With Deep Learning (2018, background): “Spectral decomposition of an enviroment’s transition function or successor function (which has the same eigenfunctions) has also been proposed by neuroscientists as a model for the emergence of grid cells in the entorhinal cortex [31].”
A map of abstract relational knowledge in the human hippocampal–entorhinal cortex (2017, background): “However, recent theoretical treatments suggest that grid-like firing might also be understood as a principal component decomposition of the covariance information between place cells (Dordek et al., 2016) or the transitions between states (Stachenfeld et al., 2016, 2014).”
Spectral Inference Networks: Unifying Deep and Spectral Learning (2018, background): “PVFs have also been proposed by neuroscientists as a model for the emergence of grid cells in the entorhinal cortex (Stachenfeld et al., 2017).”
Navigating cognition: Spatial codes for human thinking (2018, methodology): “Thereby, the grid system might provide the coordinate system for the dimensions of cognitive spaces, in line with its suggested role in extracting dimensions from task states (76, 77).”
Dynamics-aware Embeddings (2019, background): “Successor representations have been demonstrated to be an effective model of animal and human learning (Momennejad et al., 2017; Stachenfeld et al., 2017).”