A Generalist Agent
By S. Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Giménez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg and 10 more. 2022
1,117 citations recorded by the source corpus.
Inspired by progress in large-scale language modeling, we apply a similar approach towards building a single generalist agent beyond the realm of text outputs. The agent, which we refer to as Gato, works as a multi-modal, multi-task, multi-embodiment generalist policy. The same network with the same weights can play Atari, caption images, chat, stack blocks with a real robot arm and much more, deciding based on its context whether to output text, joint torques, button presses, or other tokens. In this report we describe the model and the data, and document the current capabilities of Gato.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, LLM Agents & Tool Use, Reinforcement Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Mohit Bansal
- Yoshua Bengio
- Quoc V. Le
- Jianfeng Gao
- Percy Liang
- Chelsea Finn
- Ruslan Salakhutdinov
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Jitendra Malik
- Xiaogang Wang
- Andrew Zisserman
- Ming-Hsuan Yang
- Li Fei-Fei
In LLM Agents & Tool Use
In Reinforcement Learning
- Sergey Levine
- Chelsea Finn
- Pieter Abbeel
- Shixiang Shane Gu
- Ofir Nachum
- Shimon Whiteson
- Abhishek Gupta
- Karol Hausman
Citations in the corpus
- PaLM-E: An Embodied Multimodal Language Model (2023, background): “In contrast to Gato, we demonstrate positive transfer across different tasks where the model benefits from diverse joint training across multiple domains.”
- Choose Your Weapon: Survival Strategies for Depressed AI Academics (2023, background): “From the ImageNet [19] competitions and their various winners to ChatGPT, Gato [17], and most recently to GPT-4 [1], we have seen that more data and more compute yield quantitatively and often even qualitatively better results.”
- CLiMB: A Continual Learning Benchmark for Vision-and-Language Tasks (2022, methodology): “This assumption means learning separate models for language-only, vision-only, and vision-language tasks, as opposed to a single “generalist” model that can handle all modalities or subsets of them [Reed et al., 2022].”
- Inductive biases for deep learning of higher-level cognition (2020, neutral): “with efficient transfer learning) (Ravi and Larochelle, 2016; Wang et al., 2016; Finn et al., 2017; Cabi et al., 2019; Jang et al., 2022; Reed et al., 2022; Ahn et al., 2022; Brown et al., 2020; Alayrac et al., 2022; Borgeaud et al., 2022; Chowdhery et al., 2022; Sanh et al., 2021; Lu et al., 2022; ”
- Human-Timescale Adaptation in an Open-Ended Task Space (2023, background): “2022), fine-tuning (Lee et al., 2022) or access to offline datasets (Laskin et al., 2022; Reed et al., 2022).”
- Harms from Increasingly Agentic Algorithmic Systems (2023, methodology): “For example, the raw task performance of generalist systems [Reed et al. 2022; Zeng et al. 2022] is still limited to tasks where expert data is available and has not achieved human level on all tasks.”
- GELLO: A General, Low-Cost, and Intuitive Teleoperation Framework for Robot Manipulators (2023, positive): “For manipulation, there have been great successes in leveraging imitation learning [1], [2], [3], [4] where system performance improves with larger datasets [5], [6], [7], [8].”
- ChatGPT is not all you need. A State of the Art Review of large Generative AI models (2023, background): “It works as a multi-modal, multi-task, multi-embodiment generalist policy [27].”
- Composing Ensembles of Pre-trained Models via Iterative Consensus (2022, positive): “Large pre-trained models have been widely applied to solve decisionmaking tasks, such as learning general purpose policies (Reed et al., 2022; Li et al., 2022; Shridhar et al., 2022), making planners (Huang et al.”
- Open X-Embodiment: Robotic Learning Datasets and RT-X Models : Open X-Embodiment Collaboration0 (2023, positive): “Prior work has provided initial demonstrations of X-embodiment training [27] and transfer [25, 29, 32] with transformer models.”
- Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation (2022, background): “However, Gato relies on extremely large datasets like 15K episodes for block stacking and 94K episodes for Meta-World [48] tasks.”
- Structured Pruning for Deep Convolutional Neural Networks: A Survey (2023, background): “In recent years, foundation models [214] such as GPT-3 [215] and generalist agents such as Gato [216] are the possible way to Artificial general intelligence (AGI).”
- VIMA: General Robot Manipulation with Multimodal Prompts (2022, methodology): “However, unlike our method that adopts an object-centric representation to treat individual objects as observation tokens, VIMA-Gato divides input images into patches and encodes them by a ViT (Dosovitskiy et al., 2020) to produce observation tokens.”
- Instruction-driven history-aware policies for robotic manipulations (2022, methodology): “Transformers are also used to build a multi-modal, multi-task, multi-embodiment generalist agent, GATO [77].”
- Tool Learning with Foundation Models (2023, positive): “Furthermore, current algorithms for adapting foundation models to learn specific tools generally require huge amounts of supervised data (Nakano et al., 2021; Reed et al., 2022), hindering their generalization and transferability to broader types of tools or novel situations.”
- Where to Go Next for Recommender Systems? ID- vs. Modality-based Recommender Models Revisited (2023, background): “12In fact, we notice that the NLP/CV communities are formulating most tasks into sequence learning problem with Transformer as the backbone [46, 47], e.”
- LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks (2022, background): “Note that LIFT can be applied to any generalist model with LM-like architectures, such as GATO [89].”
- Planning-oriented Autonomous Driving (2022, background): “This is a popular practice in many domains, including general vision [46, 51, 61], autonomous driving2 [8, 34, 57, 59], such as Transfuser [13], BEV-”