Imitating Interactive Intelligence

By Josh Abramson, Arun Ahuja, Arthur Brussee, Federico Carnevale, Mary Cassin, Stephen Clark, A. Dudzik, Petko Georgiev, Aurelia Guy, Tim Harley and 14 more. 2020

74 citations recorded by the source corpus.

A common vision from science fiction is that robots will one day inhabit our physical spaces, sense the world as we do, assist our physical labours, and communicate with us through natural language. Here we study how to design artificial agents that can interact naturally with humans using the simplification of a virtual environment. This setting nevertheless integrates a number of the central challenges of artificial intelligence (AI) research: complex visual perception and goal-directed physical control, grounded language comprehension and production, and multi-agent social interaction. To build agents that can robustly interact with humans, we would ideally train them while they interact with humans. However, this is presently impractical. Therefore, we approximate the role of the human with another learned agent, and use ideas from inverse reinforcement learning to reduce the disparities between human-human and agent-agent interactive behaviour. Rigorously evaluating our agents poses a great challenge, so we develop a variety of behavioural tests, including evaluation by humans who watch videos of agents or interact directly with them. These evaluations convincingly demonstrate that interactive training and auxiliary losses improve agent behaviour beyond what is achieved by supervised learning of actions alone. Further, we demonstrate that agent capabilities generalise beyond literal experiences in the dataset. Finally, we train evaluation models whose ratings of agents agree well with human judgement, thus permitting the evaluation of new agent models without additional effort. Taken together, our results in this virtual environment provide evidence that large-scale human behavioural imitation is a promising tool to create intelligent, interactive agents, and the challenge of reliably evaluating such agents is possible to surmount.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

ReAct: Synergizing Reasoning and Acting in Language Models (2022, background): “We also note that leveraging language as semantically-rich inputs in the process of interactive decision making has been shown to be successful under other settings (Abramson et al., 2020; Karamcheti et al., 2021; Huang et al., 2022a; Li et al., 2022).”
Perceiver IO: A General Architecture for Structured Inputs & Outputs (2021, background): “[1, 89, 65]), and the shape and structure of a task’s inputs and outputs may place strong constraints on how such a system processes data, making it difficult to adapt to new settings.”
Teaching Autoregressive Language Models Complex Tasks By Demonstration (2021, methodology): “In addition to the work described above, the approach taken in this case study takes inspiration from behavioral cloning [12]–[14], variants of which have been used elsewhere to leverage Transformers for imitation learning [15]–[18].”
Decision Transformer: Reinforcement Learning via Sequence Modeling (2021, background): “[66] combine language and image modalities for text-conditioned behavior generation.”
MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge (2022, methodology): “We can cache the image features of the 7 overlapping frames V[1] to V[7] to maximize compute savings.”
Language Grounding with 3D Objects (2021, background): “Such tasks can be extended to include simulated world interactions such as picking and placing objects [29] and using appliances and tools [30].”
Semantic Exploration from Language Abstractions and Pretrained Representations (2022, background): “This beneﬁt is seen across on-policy and off-policy algorithms (Impala and R2D2), different exploration methods (RND and NGU), different 3D domains (Playroom and City), and various task speciﬁcations (lifting/putting, searching, and intrinsically motivated navigation).”
Debiasing Pre-Trained Language Models via Efficient Fine-Tuning (2022, background): “Specifically in the field of NLP, fine-tuning can transfer language models such as transformers (Vaswani et al., 2017) into various other task modalities (Abramson et al., 2020; Dosovitskiy et al., 2020; Lu et al., 2021; Radford et al., 2021).”
LanCon-Learn: Learning with Language to Enable Generalization in Multi-Task Manipulation (2021, background): “These approaches, as well as more modern research [8], [21], [22], [23], [24], [25], [26], [27], [28], typically require a large corpus of commands or observations with granular instructions, as well as a detailed alignment of commands to states.”
Symbolic Behaviour in Artificial Intelligence (2021, background): “In [23], humans and agents take control of avatars in a virtual 3D environment called a “Playroom”, comprising objects such as furniture and toys.”
CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society (2023, background): “By enabling agents to function as communicators themselves, they become capable of solving complex tasks [113, 85, 72, 3, 30, 111, 79, 41, 28, 102, 80, 106, 35, 49, 2, 51, 1, 55, 50, 65, 92].”