Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration

By Xavier Puig, Tianmin Shu, Shuang Li, Zilin Wang, Josh Tenenbaum, Sanja Fidler, Antonio Torralba. 2020

185 citations recorded by the source corpus.

In this paper, we introduce Watch-And-Help (WAH), a challenge for testing social intelligence in agents. In WAH, an AI agent needs to help a human-like agent perform a complex household task efficiently. To succeed, the AI agent needs to i) understand the underlying goal of the task by watching a single demonstration of the human-like agent performing the same task (social perception), and ii) coordinate with the human-like agent to solve the task in an unseen environment as fast as possible (human-AI collaboration). For this challenge, we build VirtualHome-Social, a multi-agent household environment, and provide a benchmark including both planning and learning based baselines. We evaluate the performance of AI agents with the human-like agent as well as with real humans using objective metrics and subjective user ratings. Experimental results demonstrate that the proposed challenge and virtual environment enable a systematic evaluation on the important aspects of machine social intelligence at scale.

Subfields: Artificial Intelligence, Social Psychology, Reinforcement Learning, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Josh Tenenbaum
2. Chuang Gan
3. Yilun Du
4. David Hsu
5. Tianmin Shu
6. Zirui Zhao
7. Wee Sun Lee
8. Qinhong Zhou

In Social Psychology

1. Josh Tenenbaum
2. Tianmin Shu

In Reinforcement Learning

1. Zirui Zhao
2. Chuang Gan
3. David Hsu
4. Yilun Du
5. Josh Tenenbaum
6. Tianmin Shu
7. Qinhong Zhou
8. Hongxin Zhang

In Human-Computer Interaction

1. Yilun Du
2. Josh Tenenbaum

Citations in the corpus

• Building Cooperative Embodied Agents Modularly with Large Language Models (2023, methodology): “Although LLMs still face challenges when solving complex reasoning problems (Bubeck et al., 2023), a substantial body of work demonstrates their capacity to make plans (Sharma et al., 2021; Raman et al., 2022; Pallagani et al., 2022; Gramopadhye & Szafir, 2022; Yuan et al., 2023; Li et al., 2022; Wa”
• A Metaverse: Taxonomy, Components, Applications, and Open Challenges (2022, unknown): “Xavier et al. [197] proposed a watch-and-help (WAH) model that uses a single demonstration of an agent performing the same task to understand the task’s goal and work with a human-like agent to solve a problem.”
• ToM2C: Target-oriented Multi-agent Communication and Cooperation with Theory of Mind (2021, unknown): “(Puig et al., 2021; Carroll et al., 2019; Netanyahu et al., 2021) studies the human-AI cooperation with mental inference.”
• Large Language Models as Commonsense Knowledge for Large-Scale Task Planning (2023, methodology): “Similar to [22, 25], we maintain the belief in object-centric graphs, where nodes are objects and edges describe abstract-level relationships (e.g., apples are inside fridge, fridge is inside kitchen) between objects and rooms.”