A Generalist Agent

By S. Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Giménez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg and 10 more. 2022

1,117 citations recorded by the source corpus.

Inspired by progress in large-scale language modeling, we apply a similar approach towards building a single generalist agent beyond the realm of text outputs. The agent, which we refer to as Gato, works as a multi-modal, multi-task, multi-embodiment generalist policy. The same network with the same weights can play Atari, caption images, chat, stack blocks with a real robot arm and much more, deciding based on its context whether to output text, joint torques, button presses, or other tokens. In this report we describe the model and the data, and document the current capabilities of Gato.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, LLM Agents & Tool Use, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Sergey Levine
  2. Mohit Bansal
  3. Yoshua Bengio
  4. Quoc V. Le
  5. Jianfeng Gao
  6. Percy Liang
  7. Chelsea Finn
  8. Ruslan Salakhutdinov

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Trevor Darrell
  4. Jitendra Malik
  5. Xiaogang Wang
  6. Andrew Zisserman
  7. Ming-Hsuan Yang
  8. Li Fei-Fei

In LLM Agents & Tool Use

  1. Lucas Manuelli
  2. Dieter Fox
  3. Mohit Shridhar
  4. Adrian Weller
  5. Brian Ichter
  6. Yaodong Yang
  7. Ying Wen
  8. Andy Zeng

In Reinforcement Learning

  1. Sergey Levine
  2. Chelsea Finn
  3. Pieter Abbeel
  4. Shixiang Shane Gu
  5. Ofir Nachum
  6. Shimon Whiteson
  7. Abhishek Gupta
  8. Karol Hausman

Citations in the corpus