Reinforcement Learning in Multidimensional Environments Relies on Attention Mechanisms

By Yael Niv, R. Daniel, A. Geana, Samuel J. Gershman, Yuan Chang Leong, Angela Radulescu, Robert C. Wilson. 2015

352 citations recorded by the source corpus.

In recent years, ideas from the computational field of reinforcement learning have revolutionized the study of learning in the brain, famously providing new, precise theories of how dopamine affects learning in the basal ganglia. However, reinforcement learning algorithms are notorious for not scaling well to multidimensional environments, as is required for real-world learning. We hypothesized that the brain naturally reduces the dimensionality of real-world problems to only those dimensions that are relevant to predicting reward, and conducted an experiment to assess by what algorithms and with what neural mechanisms this “representation learning” process is realized in humans. Our results suggest that a bilateral attentional control network comprising the intraparietal sulcus, precuneus, and dorsolateral prefrontal cortex is involved in selecting what dimensions are relevant to the task at hand, effectively updating the task representation through trial and error. In this way, cortical attention mechanisms interact with learning in the basal ganglia to solve the “curse of dimensionality” in reinforcement learning.

Subfields: Cognitive Neuroscience

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

Citations in the corpus

Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “Moreover, (Niv et al., 2015) highlights the role of attention in the human brain in supporting the reinforcement learning process to address the curse of dimensionality.”
Model-based predictions for dopamine (2017, neutral): “However, apart from the challenges that timing poses to such an account, even straightforward neural representations of the environment are an interpretation of the external reality through, at minimum, a relevance filter [29,30].”
Human Orbitofrontal Cortex Represents a Cognitive Map of State Space. (2016, background): “As a result, a task should be represented with a small but sufficient set of states, implying that only task-relevant state information should be encoded in the OFC, at least to the extent that the animal has enough familiarity with the task to know the correct state representation (Gershman and Niv”
Reinforcement learning, efficient coding, and the statistics of natural tasks (2015, background): “Such causal sparsity gives rise to ample opportunities for efficient coding [49,61].”