Open-Ended Learning Leads to Generally Capable Agents

By OpenLLMAI Team, Adam Stooke, Anuj Mahajan, C. Barros, Charlie Deck, Jakob Bauer, Jakub Sygnowski, Maja Trebacz, Max Jaderberg, Michaël Mathieu and 8 more. 2021

230 citations recorded by the source corpus.

In this work we create agents that can perform well beyond a single, individual task, that exhibit much wider generalisation of behaviour to a massive, rich space of challenges. We define a universe of tasks within an environment domain and demonstrate the ability to train agents that are generally capable across this vast space and beyond. The environment is natively multi-agent, spanning the continuum of competitive, cooperative, and independent games, which are situated within procedurally generated physical 3D worlds. The resulting space is exceptionally diverse in terms of the challenges posed to agents, and as such, even measuring the learning progress of an agent is an open research problem. We propose an iterative notion of improvement between successive generations of agents, rather than seeking to maximise a singular objective, allowing us to quantify progress despite tasks being incomparable in terms of achievable rewards. We show that through constructing an open-ended learning process, which dynamically changes the training task distributions and training objectives such that the agent never stops learning, we achieve consistent learning of new behaviours. The resulting agent is able to score reward in every one of our humanly solvable evaluation levels, with behaviour generalising to many held-out points in the universe of tasks. Examples of this zero-shot generalisation include good performance on Hide and Seek, Capture the Flag, and Tag. Through analysis and hand-authored probe tasks we characterise the behaviour of our agent, and find interesting emergent heuristic behaviours such as trial-and-error experimentation, simple tool use, option switching, and cooperation. Finally, we demonstrate that the general capabilities of this agent could unlock larger scale transfer of behaviour through cheap finetuning.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Augmented Language Models: a Survey (2023, methodology): “…with an environment, and it has been used for applications such as playing games (Mnih et al., 2015; Silver et al., 2016; Vinyals et al., 2019; Team et al., 2021; Bakhtin et al., 2022) or controlling robots (Gu et al., 2017; Kalashnikov et al., 2018; Akkaya et al., 2019; Lee et al., 2020).”
Improving alignment of dialogue agents via targeted human judgements (2022, background): “A long-term approach should make the trade-off of helpfulness and harmlessness test-time configurable (Abdolmaleki et al., 2020) and train over an expanding universe of trade-offs and topics in an open ended fashion (Open Ended Learning Team et al., 2021) to find an optimal training data distributio”
Disentangling Transfer in Continual Reinforcement Learning (2022, background): “[44, 4] advocate the need to develop CL methods suitable for reinforcement learning training as a necessary step towards learning artificial intelligent agents to operate in open-ended and changing environments.”
TiKick: Towards Playing Multi-agent Football Full Games from Single-agent Demonstrations (2021, neutral): “However, current DRL systems still suffer from challenges of multi-agent coordination (Rashid et al., 2018; Mahajan et al., 2019; Yu et al., 2021), sparse rewards (Taiga et al., 2019; Zhang et al., 2020), stochastic environments (Kurach et al., 2019; Team et al., 2021), etc.”
VIMA: General Robot Manipulation with Multimodal Prompts (2022, background): “, 2021) is adopting the large-scale pre-training paradigm, powered by a collection of learning environments (Abramson et al., 2020; Shridhar et al., 2020; Savva et al., 2019; Puig et al., 2018; Team et al., 2021; Toyama et al., 2021; Shi et al., 2017).”
Foundation models in robotics: Applications, challenges, and the future (2023, unknown): “The value function is implicitly deﬁned via distance embedding.”
Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory (2023, background): “Given its massive scale, vast success, and unrestricted freedom, Minecraft has established itself as an unparalleled platform for researching autonomous and robust Generally Capable Agents (GCAs) [23] in open-world environments brimmed with long-horizon challenges, environmental disruptions, and unc”
Symbolic Behaviour in Artificial Intelligence (2021, background): “Furthermore, generalization can be qualitatively different when experience is more interactive or embodied [43], and training tasks are more diverse [101].”
ProcTHOR: Large-Scale Embodied AI Using Procedural Generation (2022, background): “10 [2] Adam, Anuj Mahajan, Catarina Barros, Charlie Deck, Jakob Bauer, Jakub Sygnowski, Maja Trebacz, Max Jaderberg, Michael Mathieu, Nat McAleese, Nathalie Bradley-Schmieg, Nathaniel Wong, Nicolas Porcel, Roberta Raileanu, Steph Hughes-Fitt, Valentin Dalibard, and Wojciech Marian Czarnecki.”
Vygotskian Autotelic Artificial Intelligence: Language and Culture Internalization for Human-Like AI (2022, background): “14,17,62,63 It will not cover the advances in the sub-ﬁeld of rl dedicated to the computational modeling of social interactions (social rl ).”
Genie: Generative Interactive Environments (2024, background): “…from unseen action-free videos of simulated reinforcement learning (RL) environments, indicating that Genie may hold the key to unlocking unlimited data for training the next generation of generalist agents (Bauer et al., 2023; Clune, 2019; Open Ended Learning Team et al., 2021; Reed et al., 2022).”
Human-Timescale Adaptation in an Open-Ended Task Space (2023, methodology): “…distillation loss (Schmidhuber, 1992; Schmitt et al., 2018; Czarnecki et al., 2019) to guide AdA’s learning with the policy of a pre-trained teacher, in a process known as kickstarting; iterating this process leads to a generational training regime (OEL Team et al., 2021; Wang et al., 2021).”
L2Explorer: A Lifelong Reinforcement Learning Assessment Environment (2022, background): “These include the open-world game environments of XLand (Team et al. 2021) and MiniHack (Samvelyan et al. 2021).”
The alignment problem from a deep learning perspective (2022, background): “…and Diplomacy [Bakhtin et al., 2022], while also exhibiting growing generality [Bom-masani et al., 2021] through improvements in areas including sample efﬁciency [Brown et al., 2020, Dorner, 2021], cross-task generalization [Adam et al., 2021], and multi-step reasoning [Chowdhery et al., 2022].”
Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos (2022, background): “[83] Open Ended Learning Team, Adam Stooke, Anuj Mahajan, Catarina Barros, Charlie Deck, Jakob Bauer, Jakub Sygnowski, Maja Trebacz, Max Jaderberg, Michael Mathieu, et al.”
MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge (2022, methodology): “Prior works include maze-style worlds [121, 129, 61], purely text-based game [69], grid worlds [21, 16], browser/GUI-based environments [108, 124], and indoor simulators for robotics [1, 107, 114, 34, 110, 99, 89].”
LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning (2023, methodology): “Besides supervised learning datasets, video game benchmarks (e.g., Atari [46], XLand [64], and VisDoom [30]) in reinforcement learning (RL) have also been used for studying LL.”
When LLMs Play the Telephone Game: Cultural Attractors as Conceptual Tools to Evaluate LLMs in Multi-turn Settings (2024, unknown): “Transmission chains featuring artificial agents Several studies have applied experimental designs used in cultural evolution to study knowledge and skill accumulation in groups of Reinforcement Learning agents (Cook et al., 2024; Schmitt et al., 2018; Team et al., 2021; Prystawski et al., 2023).”