Learning latent structure: Carving nature at its joints

By Samuel J. Gershman, Yael Niv. Current Opinion in Neurobiology, 2010

328 citations recorded by the source corpus.

Reinforcement learning algorithms provide powerful explanations for simple learning and decision making behaviors and the functions of their underlying neural substrates. Unfortunately, in real world situations that involve many stimuli and actions, these algorithms learn pitifully slowly, exposing their inferiority in comparison to animal and human learning. Here we suggest that one reason for this discrepancy is that humans and animals take advantage of structure that is inherent in real-world tasks to simplify the learning problem. We survey an emerging literature on “structure learning”—using experience to infer the structure of a task—and how this can be of service to reinforcement learning, with an emphasis on structure in perception and action.

Subfields: Cognitive Neuroscience, Social Psychology

View Paper: https://www.semanticscholar.org/paper/0b376fd75196486fd1d32bfb7bd62dcae89f0cbb

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Social Psychology

Citations in the corpus

Human Orbitofrontal Cortex Represents a Cognitive Map of State Space. (2016, background): “As a result, a task should be represented with a small but sufficient set of states, implying that only task-relevant state information should be encoded in the OFC, at least to the extent that the animal has enough familiarity with the task to know the correct state representation (Gershman and Niv”
Orbitofrontal cortex as a cognitive map of task space (2014, unknown): “Thus, our representation generalizes to probabilistic reversal learning tasks (e.g., Tsuchida et al., 2010) in which the animal (and model) must infer what state it is in by using actions and outcomes from multiple previous trials (Gershman et al., 2010).”
Active Inference, Curiosity and Insight (2017, unknown): “In other words, we have avoided many important questions about the construction and exploration of model spaces in the absence of a full model (Gershman & Niv, 2010; Navarro & Perfors, 2011; Collins & Frank, 2013; Tervo et al., 2016).”
Exploration from Generalization Mediated by Multiple Controllers (2013, background): “From the perspective of model-based RL, the subjects may be building a causal description of the original environment (Courville et al. 2004; Gershman et al. 2010b; Tenenbaum et al. 2006) indicating what could and could not be responsible for their dismal fate.”
Attention, learning and the value of information (2012, background): “…studies makes it clear that processes of information selection tap into some of our highest cognitive functions, involving, among others, intrinsic curiosity and the ability for advance planning and forming internal models of complex tasks (e.g., Gershman and Niv, 2010; Johnson et al., 2012).”
Perception, Action, and Utility: The Tangled Skein (2012, background): “Indeed, numerous authors (Daw et al., 2006; Dayan and Daw, 2008; Braun et al., 2010; Gershman and Niv, 2010; Rao, 2010) have argued that the full problem of decision making under perceptual uncertainty—when both the perceptual state and the utilities are unknown and non-trivial—can essentially be tr”
Model-based predictions for dopamine (2017, unknown): “The mapping between observations (such as cues and rewards) and underlying task states may be probabilistic (as in ‘partially observable environments’) or ambiguous (for example in the case of conﬂicting or mixed cues) [19,35–37], making state inference itself a non-trivial process.”
The Tolman-Eichenbaum Machine: Unifying Space and Relational Memory through Generalization in the Hippocampal Formation (2019, background): “Situations where sensory input is the same, but task location is different are referred to as latent states in reinforcement learning (Gershman and Niv, 2010).”
Human Neuroscience (unknown): “These practical limitations have led some authors to argue that Bayesian principles and " structure learning " are essential in improving RL approaches (Gershman and Niv, 2010).”
Reinforcement Learning and Causal Models (2017, background): “This is a form of latent structure learning (Courville, Daw, & Touretzky, 2006; Gershman & Niv, 2010).”
Reinforcement Learning in Multidimensional Environments Relies on Attention Mechanisms (2015, unknown): “This process has been termed “representation learning” as it is tantamount to selecting a suitable state representation for the task (Gershman and Niv, 2010; Wilson and Niv, 2011).”
Cognitive Control Predicts Use of Model-based Reinforcement Learning (2014, methodology): “Recent research (Collins & Frank, 2013; Collins & Koechlin, 2012; Gershman & Niv, 2010) aims to extend principles of RL, hierarchically, to learning at this level.”
Simple Plans or Sophisticated Habits? State, Transition and Learning Interactions in the Two-Step Task (2015, background): “These considerations bring to the forefront the question of what state representations are learned and used [53,54], something known to be central to the speed with which agents learn to solve decision problems.”
Discovering latent causes in reinforcement learning (2015, background): “Recent work has begun to unravel the cognitive and neural mechanisms supporting structure learning (see [2,3] for reviews).”