Reinforcement Learning, Fast and Slow.

By Matthew Botvinick, Samuel Ritter, Jane X. Wang, Zeb Kurth-Nelson, Charles Blundell, Demis Hassabis. 2019

659 citations recorded by the source corpus.

Deep reinforcement learning (RL) methods have driven impressive advances in artificial intelligence in recent years, exceeding human performance in domains ranging from Atari to Go to no-limit poker. This progress has drawn the attention of cognitive scientists interested in understanding human learning. However, the concern has been raised that deep RL may be too sample-inefficient - that is, it may simply be too slow - to provide a plausible model of how humans learn. In the present review, we counter this critique by describing recently developed techniques that allow deep RL to operate more nimbly, solving problems much more quickly than previous methods. Although these techniques were developed in an AI context, we propose that they may have rich implications for psychology and neuroscience. A key insight, arising from these AI methods, concerns the fundamental connection between fast RL and slower, more incremental forms of learning.

Subfields: Cognitive Neuroscience, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Reinforcement Learning

Citations in the corpus

Environmental drivers of systematicity and generalization in a situated agent (2019, background): “It is also worth noting that none of our experiments reﬂect the human ability to learn (rather than extend existing knowledge) quickly (as in, e.g. Lake et al. (2015); Botvinick et al. (2019)).”
Toward a Brain-Inspired System: Deep Recurrent Reinforcement Learning for a Simulated Self-Driving Agent (2019, background): “In recent years, some bio-inspired intelligent methods have emerged (Marblestone et al., 2016; Gershman and Daw, 2017; Hassabis et al., 2017; Botvinick et al., 2019), and they are clearly different from the classical mathematical programming principle.”
The Role of Executive Function in Shaping Reinforcement Learning. (2020, background): “This formalized account of learning and decision making has contributed significantly to expanding the frontiers of artificial intelligence research [2], our understanding of clinical pathologies [3,4], and research on developmental changes in learning [5,6].”
DREAM Architecture: a Developmental Approach to Open-Ended Learning in Robotics (2020, neutral): “The features of an MDP representation constrains the kind of learning algorithms that can be used: a small and discrete set of actions and states facilitates an exhaustive exploration to discover the most relevant policy [3], while a large dimension and continuous state and action spaces raise explo”
Fast Adaptive Task Offloading in Edge Computing Based on Meta Reinforcement Learning (2020, background): “Based on the meta policy, the “inner loop” can adapt fast to new tasks through a small number of gradient updates [15].”
Unsupervised Curricula for Visual Meta-Reinforcement Learning (2019, methodology): “Hence, these algorithms still rely on slow RL [7] in order to adapt to new tasks posed at test-time.”
Rapid Task-Solving in Novel Environments (2020, background): “This can be attributed to their reliance on gradient descent for encoding knowledge and their inability to bring prior knowledge to new environments (Botvinick et al., 2019).”
Integrating machine learning and multiscale modeling—perspectives, challenges, and opportunities in the biological, biomedical, and behavioral sciences (2019, neutral): “[6] Botvinick, M.”
Deep reinforcement learning: a survey (2020, background): “Botvinick et al. (2019) summarized the implementation of deep meta-RL and proposed the future directions like combining episodic deep RL with meta-RL.”
Fast and Slow Learning of Recurrent Independent Mechanisms (2021, background): “…rapidly with a few training examples, and has demonstrated success in both supervised learning such as few shot image classiﬁcation (Ravi & Larochelle, 2016) and reinforcement learning (Wang et al., 2016; Santoro et al., 2016; Duan et al., 2016; Botvinick et al., 2019; Clune, 2019) settings.”
Direct-fit to nature: an evolutionary perspective on biological (and artificial) neural networks (2019, background): “However, in practice, interpretability and generalization are often at odds: interpretable models may have considerable explanatory appeal but poor predictive power, while high-performing predictive models may be difficult to interpret (Breiman, 2001; Shmueli, 2010; Yarkoni and Westfall, 2017).”
Distributional reinforcement learning in the brain (2020, background): “Rearranging and sampling rt ~ R(st) from the environment, we arrive at a new kind of RPE, namely, a TD error [4], which we also call δ to emphasize its similarity to the RPE in the RW model:”
Artificial intelligence in recommender systems (2020, background): “Thegeneralityofreinforce-ment learning makes it widely applied in various aspects such as game theory [50], optimal control [51], swarm intelligence [52] and other areas such as healthcare [53] and psychology [54].”
Emergent Systematic Generalization in a Situated Agent (2019, background): “…in no way encompass the full range of systematicity of thought/behaviour that one might expect of a mature adult human, and that none of our experiments reﬂect the human ability to learn (rather than extend existing knowledge) quickly (as in, e.g. Lake et al. (2015); Botvinick et al. (2019)).”