Prefrontal cortex as a meta-reinforcement learning system

By Jane X. Wang, Zeb Kurth-Nelson, Dharshan Kumaran, Dhruva Tirumala, Hubert Soyer, Joel Z. Leibo, Demis Hassabis, Matthew Botvinick. 2018

627 citations recorded by the source corpus.

Subfields: Cognitive Neuroscience, Experimental and Cognitive Psychology, Reinforcement Learning, Neuroscience

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Experimental and Cognitive Psychology

In Reinforcement Learning

In Neuroscience

Citations in the corpus

Causal Reasoning from Meta-reinforcement Learning (2019, methodology): “For example, in previous work this approach was used to negotiate the exploration-exploitation tradeoff in multi-armed bandits (Duan et al., 2016) and learn algorithms which dynamically adjust their own learning rates (Wang et al., 2016; 2018).”
Relational inductive biases, deep learning, and graph networks (2018, methodology): “…et al., 2015; Gilmer et al., 2017), to predict traﬃc on roads (Li et al., 2017; Cui et al., 2018), to classify and segment images and videos (Wang et al., 2018c; Hu et al., 2017) and 3D meshes and point clouds (Wang et al., 2018d), to classify regions in images (Chen et al., 2018a), to…”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “It has also been theorized that the dopamine system is directly responsible for a kind of meta-learning by training the prefrontal cortex, which then can function as its own standalone learning system (Wang et al., 2018).”
A modeling framework for adaptive lifelong learning with transfer and savings through gating in the prefrontal cortex (2020, methodology): “(9) (for details, see SI Appendix, Supporting Information Text).”
The Tools Challenge: Rapid Trial-and-Error Learning in Physical Problem Solving (2019, positive): “…as an exciting direction for tackling multi-task reinforcement learning in model-based and model-free settings (Finn, Abbeel, & Levine, 2017; Rakelly, Zhou, Finn, Levine, & Quillen, 2019; Gupta, Mendonca, Liu, Abbeel, & Levine, 2018; Schmidhuber, Zhao, & Schraudolph, 1998; Wang et al., 2018) .”
Rapid trial-and-error learning with simulation supports flexible tool use and physical reasoning (2019, background): “ model-free settings (Finn, Abbeel, & Levine, 2017; Rakelly, Zhou, Finn, Levine, & Quillen, 2019; Gupta, Mendonca, Liu, Abbeel, & Levine, 2018; Schmidhuber, Zhao, & Schraudolph, 1998; Wang et al., 2018) . While such approaches seem promising for the Tools game, they have mostly been ”
What Is a Cognitive Map? Organizing Knowledge for Flexible Behavior. (2018, background): “Dopamine signals a reward prediction error that is assumed to cause learning but, in prefrontal cortex at least, triggers synaptic change over thewrong timescale—tens of seconds at the fastest (Brzosko et al., 2015; Otmakhova and Lisman, 1996; Wang et al., 2018; Yagishita et al., 2014), when behavio”
Multimodal Model-Agnostic Meta-Learning via Task-Aware Modulation (2019, methodology): “On the other hand, instead of learning a metric, model-based (i.e. RNN-based) meta-learning models learn to adjust model states (e.g. a state of an RNN [25, 4, 51] or external memory [37, 27]) using a training dataset and output the parameters of a learned model or the predictions given test inputs.”
An investigation of model-free planning (2019, background): “Previous work (Pang & Werbos, 1998; Wang et al., 2018) has supported the idea that model-based behavior can be learned with general recurrent architectures, with planning computation amortized over multiple discrete steps (Schmidhuber, 1990), but comprehensive demonstrations of its effectiveness are”
Backpropamine: training self-modifying neural networks with differentiable neuromodulated plasticity (2018, methodology): “…can compute (reward predictions, surprise, saliency, etc.) This emergent weight-modifying algorithm (designed over many episodes/lifetimes by the “outer loop” meta-training algorithm) might in turn sculpt network connectivity to implement the meta-learning process described by Wang et al. (2018).”
Long short-term memory and Learning-to-learn in networks of spiking neurons (2018, background): “In addition, [8] demonstrates that meta-RL provides a very interesting perspective of reward-based learning in the brain.”
The effects of negative adaptation in Model-Agnostic Meta-Learning (2018, background): “The notion of fast learning and adaptation in meta-reinforcement learning can also be motivated from a neuroscience and psychology point of view [17].”
Human Replay Spontaneously Reorganizes Experience (2019, background): “This can be viewed as ‘‘meta-learning’’ (Harlow, 1949; Shin et al., 2017; Wang et al., 2018), insomuch as previous learning facilitated rapid integration of new knowledge.”
Deep Reinforcement Learning (2018, background): “ng (NLP).Xu et al.(2015) integrate attention to image captioning. See alsoLiu et al.(2016),Lu et al.(2016),Rennie et al. (2017), andRen et al.(2017) for image captioning. SeePasunuru and Bansal(2017);Wang et al. (2018f) for video captioning.Strub et al.(2017) propose end-to-end optimization with dee”
Reinforcement Learning, Fast and Slow. (2019, positive): “[49] proposed that meta-learning over the course of an animal's lifetime shapes the recurrent networks in PFC to have tuning properties that are useful for naturalistic learning tasks.”
Human-Timescale Adaptation in an Open-Ended Task Space (2023, background): “This has proven to be an effective approach that can learn Bayes-optimal strategies (Ortega et al., 2019; Mikulik et al., 2020) and may have neurological analogues (Wang et al., 2018).”