The successor representation in human reinforcement learning

By Ida Momennejad, Evan M. Russek, J. H. Cheong, Matthew Botvinick, Nathaniel D. Daw, Samuel J. Gershman. 2016

409 citations recorded by the source corpus.

Theories of reward learning in neuroscience have focused on two families of algorithms thought to capture deliberative versus habitual choice. ‘Model-based’ algorithms compute the value of candidate actions from scratch, whereas ‘model-free’ algorithms make choice more efficient but less flexible by storing pre-computed action values. We examine an intermediate algorithmic family, the successor representation, which balances flexibility and efficiency by storing partially computed action values: predictions about future events. These pre-computation strategies differ in how they update their choices following changes in a task. The successor representation’s reliance on stored predictions about future states predicts a unique signature of insensitivity to changes in the task’s sequence of events, but flexible adjustment following changes to rewards. We provide evidence for such differential sensitivity in two behavioural studies with humans. These results suggest that the successor representation is a computational substrate for semi-flexible choice in humans, introducing a subtler, more cognitive notion of habit. Momennejad et al. formulate and provide evidence for the successor representation, a computational learning mechanism intermediate between the two dominant models (a fast but inflexible ‘model-free’ system and a flexible but slow ‘model-based’ one).

Subfields: Cognitive Neuroscience

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

Citations in the corpus

Towards Continual Reinforcement Learning: A Review and Perspectives (2020, positive): “that the successor representation model from the RL literature (Dayan, 1993) may provide a more accurate model to reflect the way humans balance their use of both model-based and model-free learning (Momennejad et al., 2017).”
Dynamics-aware Embeddings (2019, background): “Successor representations have been demonstrated to be an effective model of animal and human learning (Momennejad et al., 2017; Stachenfeld et al., 2017).”
A map of abstract relational knowledge in the human hippocampal–entorhinal cortex (2017, background): “…may facilitate reinforcement learning, because the resulting predictive measure of future states could be flexibly combined with reward representations to enable rapid computation of navigational trajectories (Baram et al., 2017; Dayan, 1993; Momennejad et al., 2016; Russek et al., 2016).”
The role of prefrontal cortex in cognitive control and executive function (2021, positive): “A recent computational formulation [186] has suggested an intermediate type of control mode relying on model-based and modelfree computations guided by “successor representations’“ that enable behavior to be both flexibly goal-directed but also efficiently model free.”
Offline replay supports planning in human reinforcement learning (2018, methodology): “, 2017), computational models suggest that Dyna-like offline replay mechanisms might be implemented in the brain (Momennejad et al., 2017; Johnson and Redish, 2005; Ludvig et al., 2017; Russek et al., 2017).”
γ-Models: Generative Temporal Difference Learning for Infinite-Horizon Prediction (2020, background): “The most closely related prior work to -models is the successor representation (Dayan, 1993), a formulation of long-horizon prediction that has been influential in both cognitive science (Momennejad et al., 2017; Gershman, 2018) and machine learning (Kulkarni et al.”