Model-Based and Model-Free Pavlovian Reward Learning: Revaluation, Revision and Revelation

By Peter S. Dayan, Kent C. Berridge. 2014

331 citations recorded by the source corpus.

Evidence supports at least two methods for learning about reward and punishment and making predictions for guiding actions. One method, called model-free, progressively acquires cached estimates of the long-run values of circumstances and actions from retrospective experience. The other method, called model-based, uses representations of the environment, expectations, and prospective calculations to make cognitive predictions of future value. Extensive attention has been paid to both methods in computational analyses of instrumental learning. By contrast, although a full computational analysis has been lacking, Pavlovian learning and prediction has typically been presumed to be solely model-free. Here, we revise that presumption and review compelling evidence from Pavlovian revaluation experiments showing that Pavlovian predictions can involve their own form of model-based evaluation. In model-based Pavlovian evaluation, prevailing states of the body and brain influence value computations, and thereby produce powerful incentive motivations that can sometimes be quite new. We consider the consequences of this revised Pavlovian view for the computational landscape of prediction, response, and choice. We also revisit differences between Pavlovian and instrumental learning in the control of incentive motivation.

Subfields: Cognitive Neuroscience, Experimental and Cognitive Psychology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Experimental and Cognitive Psychology

Citations in the corpus

Adaptive Online Planning for Continual Lifelong Learning (2019, background): “When uncertain about a course of action, humans use an elongated model-based search to evaluate long-term trajectories, but fall back on habitual behaviors learned with model-free paradigms when they are certain of what to do [3, 8, 9, 16].”
Model-based predictions for dopamine (2017, unknown): “Another important feature of model-free learning (including TDRL) is that it posits that scalar state values are accrued solely through experiencing the relationship between the current state and the (possibly rewarded) state that follows [6,16].”
Pleasure systems in the brain (2015, background): “That’s because it excludes any dynamic modulation of incentive salience by shifts in physiological states, which often occurs in real life and which would permit experimental separation of learned versus motivation values (Berridge, 2012; Dayan and Berridge, 2014; Robinson and Berridge, 2013).”
How Computational Modeling Can Force Theory Building in Psychological Science (2020, background): “Some frameworks appear superficially simple or narrow, such as the concept of working memory (Baddeley, 2010) or dual-systems approaches (Dayan & Berridge, 2014; Kahneman, 2011), whereas others can be all-encompassing such as unified theories of cognition (Newell, 1990) or connectionism (McClelland…”