Two Body Problem: Collaborative Visual Task Completion

By Unnat Jain, Luca Weihs, Eric Kolve, Mohammad Rastegari, Svetlana Lazebnik, Ali Farhadi, Alexander Gerhard Schwing, Aniruddha Kembhavi. 2019

76 citations recorded by the source corpus.

Collaboration is a necessary skill to perform tasks that are beyond one agent's capabilities. Addressed extensively in both conventional and modern AI, multi-agent collaboration has often been studied in the context of simple grid worlds. We argue that there are inherently visual aspects to collaboration which should be studied in visually rich environments. A key element in collaboration is communication that can be either explicit, through messages, or implicit, through perception of the other agents and the visual world. Learning to collaborate in a visual environment entails learning (1) to perform the task, (2) when and what to communicate, and (3) how to act based on these communications and the perception of the visual world. In this paper we study the problem of learning to collaborate directly from pixels in AI2-THOR and demonstrate the benefits of explicit and implicit modes of communication to perform visual tasks. Refer to our project page for more details: https://prior.allenai.org/projects/two-body-problem

Subfields: Aerospace Engineering, Artificial Intelligence, Mechanical Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Artificial Intelligence

In Mechanical Engineering

Citations in the corpus

AI2-THOR: An Interactive 3D Environment for Visual AI (2017, background): “Some other notable multi-agent work includes [35], which tasks agents with playing Cache, a variant of hide-and-seek where one agent hides an object and the other agent is tasked with finding that object; [33], which uses multiple agents for interactive question answering; [20], which proposes using”
Who2com: Collaborative Perception via Learnable Handshake Communication (2020, background): “[15], studying collaboration in simplistic environments does not permit study of the interplay between perception and communication.”
A Cordial Sync: Going Beyond Marginal Policies for Multi-Agent Embodied Tasks (2020, methodology): “Visual, multi-agent, collaborative tasks have not been studied until very recently [23,42].”
When2com: Multi-Agent Perception via Communication Graph Grouping (2020, background): “As mentioned in Jain et al. [16], studying agents’ interactions in simplistic environments does not permit study of the inter-play between perception and communication.”
AllenAct: A Framework for Embodied AI Research (2020, methodology): “…[18, 19], language grounding [20, 21], grasping [22, 23], object manipulation [24, 25], future prediction [26], and multi-agent collaboration [27, 28]; as well as using interaction as a tool to learn: environment representations [29], intuitive physics [30], and objects and attributes [31].”
SoundSpaces: Audio-Visual Navigation in 3D Environments (2019, methodology): “Agents are allowed a time horizon of 500 actions for all tasks, similar to [59,47,14,37,52].”
Visual Room Rearrangement (2021, background): “…this domain: navigation towards objects [4, 51, 48, 7] or towards a speciﬁc point [1, 38, 47, 36], scene exploration [9, 8], embodied question answering [18, 13], task completion [55], instruction following [2, 41], object manipulation [16, 52], multi-agent coordination [24, 23], and many others.”
Affordances from Human Videos as a Versatile Representation for Robotics (2023, background): “Exploration methods are usually guided by intrinsic rewards that are selfgenerated by the robotic agent, and are not specific to any task [9, 49, 51, 64, 73, 86, 90, 93, 98, 117].”
Shaping embodied agent behavior with activity-context priors from egocentric video (2021, methodology): “Prior approaches use imitation learning (IL) based pre-training to improve sample efﬁciency, but at the expense of collecting expert in-domain demonstrations individually for each task [68, 15, 30, 58].”
Multi-agent deep reinforcement learning: a survey (2021, background): “Jain et al. (2019) proposed TBONE for visual navigation in cooperative tasks.”
ProcTHOR: Large-Scale Embodied AI Using Procedural Generation (2022, background): “40 [52] Unnat Jain, Luca Weihs, Eric Kolve, Mohammad Rastegari, Svetlana Lazebnik, Ali Farhadi, Alexander G Schwing, and Aniruddha Kembhavi.”
Audio-Visual Embodied Navigation (2019, methodology): “Visual navigation can be tied with other tasks to attain intelligent behavior, such as question answering [23, 25, 39, 99], active visual recognition [50, 110, 109], instruction following [88, 6, 86, 17, 16] and collaborative tasks with visual inputs [49, 48, 24].”
Building Cooperative Embodied Agents Modularly with Large Language Models (2023, background): “Although LLMs still face challenges when solving complex reasoning problems [6], a substantial body of work demonstrates their capacity to assist agents in planning [41, 37, 31, 13, 52, 53], especially in embodied environments[23, 4, 30, 24, 42, 29, 54, 5, 50, 40, 51, 18, 19].”