Human-Timescale Adaptation in an Open-Ended Task Space

By A. Team, Jakob Bauer, Kate Baumli, Satinder Baveja, Feryal M. P. Behbahani, Avishkar Bhoopchand, Nathalie Bradley-Schmieg, Michael Chang, Natalie Clay, Adrian Collister and 18 more. 2023

163 citations recorded by the source corpus.

Foundation models have shown impressive adaptation and scalability in supervised and self-supervised learning problems, but so far these successes have not fully translated to reinforcement learning (RL). In this work, we demonstrate that training an RL agent at scale leads to a general in-context learning algorithm that can adapt to open-ended novel embodied 3D problems as quickly as humans. In a vast space of held-out environment dynamics, our adaptive agent (AdA) displays on-the-fly hypothesis-driven exploration, efficient exploitation of acquired knowledge, and can successfully be prompted with first-person demonstrations. Adaptation emerges from three ingredients: (1) meta-reinforcement learning across a vast, smooth and diverse task distribution, (2) a policy parameterised as a large-scale attention-based memory architecture, and (3) an effective automated curriculum that prioritises tasks at the frontier of an agent's capabilities. We demonstrate characteristic scaling laws with respect to network size, memory length, and richness of the training task distribution. We believe our results lay the foundation for increasingly general and adaptive RL agents that perform well across ever-larger open-ended domains.

Subfields: Artificial Intelligence, Reinforcement Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

In Meta & Continual Learning

Julian Togelius

Citations in the corpus

Choose Your Weapon: Survival Strategies for Depressed AI Academics (2023, background): “Procedural content generation for games was a very niche topic 15 years ago and we helped nuild a research community around it [24], [28]; recently it has become more important not only for the games industry, but also as a way to help generalize (deep) reinforcement learning [18], [23].”
Harms from Increasingly Agentic Algorithmic Systems (2023, background): “One particular example to highlight is AdA [181], which adapts to”
Foundation models in robotics: Applications, challenges, and the future (2023, unknown): “In Adaptive Agent (AdA) [30] the authors present an RL foundation model that is an agent pretrained on diverse tasks and is designed to quickly adapt to open-ended embodied 3D problems by using fast in-context learning from feedback.”