Human Reinforcement Learning Subdivides Structured Action Spaces by Learning Effector-Specific Values

By Samuel J. Gershman, Bijan Pesaran, Nathaniel D. Daw. 2009

126 citations recorded by the source corpus.

Humans and animals are endowed with a large number of effectors. Although this enables great behavioral flexibility, it presents an equally formidable reinforcement learning problem of discovering which actions are most valuable because of the high dimensionality of the action space. An unresolved question is how neural systems for reinforcement learning—such as prediction error signals for action valuation associated with dopamine and the striatum—can cope with this “curse of dimensionality.” We propose a reinforcement learning framework that allows for learned action valuations to be decomposed into effector-specific components when appropriate to a task, and test it by studying to what extent human behavior and blood oxygen level-dependent (BOLD) activity can exploit such a decomposition in a multieffector choice task. Subjects made simultaneous decisions with their left and right hands and received separate reward feedback for each hand movement. We found that choice behavior was better described by a learning model that decomposed the values of bimanual movements into separate values for each effector, rather than a traditional model that treated the bimanual actions as unitary with a single value. A decomposition of value into effector-specific components was also observed in value-related BOLD signaling, in the form of lateralized biases in striatal correlates of prediction error and anticipatory value correlates in the intraparietal sulcus. These results suggest that the human brain can use decomposed value representations to “divide and conquer” reinforcement learning over high-dimensional action spaces.

Subfields: Cognitive Neuroscience

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

Citations in the corpus

Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “There is also significant evidence that substantial hierarchical structure is present in the human learning process (Botvinick, Niv, and Barto, 2009; Gershman, Pesaran, and Daw, 2009; Frank and Badre, 2012; Badre and Frank, 2012; Diuk, Schapiro, Córdova, Ribas-Fernandes, Niv, and Botvinick, 2013;…”
Generalization of value in reinforcement learning by humans (2012, methodology): “Accordingly (following previous work: Daw et al., 2006; Schönberg et al., 2007; Gershman et al., 2009), we generated regressors for each participant using a single setting of the reinforcement learning model’s free parameters, here taken as the mean over all participants of the best-fitting…”
Reinforcement learning, efficient coding, and the statistics of natural tasks (2015, background): “[48] studied behavior and brain activation in a task where”
Learning latent structure: Carving nature at its joints (2010, background): “Gershman, Pesaran and Daw [50] designed a reinforcement learning task in which subjects were asked to make two choices simultaneously, one with each hand, after which probabilistic reward feedback was presented.”
Mechanisms of hierarchical reinforcement learning in corticostriatal circuits 1: computational analysis. (2012, background): “This is intended to reﬂect the hypothesized mechanism in which dopaminergic prediction error signals are selectively ampliﬁed in striatal cells that generated the preceding action (Frank 2005; O’Reilly and Frank 2006) and is consistent with recent functional imaging data (Gershman et al. 2009).”
Mechanisms of hierarchical reinforcement learning in cortico-striatal circuits 2: evidence from fMRI. (2012, background): “…prior evidence from reinforcement learning has routinely located activation in medial and orbital frontal regions along with striatal regions in association with RPE during reinforcement learning tasks (McClure et al. 2003; O'Doherty et al. 2004; Gershman et al. 2009; Rutledge et al. 2010).”
States versus Rewards: Dissociable neural prediction error signals underlying model-based and model-free reinforcement learning (2010, methodology): “A single set of parameters, as frequently employed in our recent work (Daw et al., 2006; Gershman et al., 2009; Gläscher et al., 2009) imposes a simple but efficient regularization that stabilizes the estimated model parameters.”