On the Utility of Learning about Humans for Human-AI Coordination

By Micah Carroll, Rohin Shah, Mark K. Ho, T. Griffiths, Sanjit A. Seshia, Pieter Abbeel, Anca Dragan. Neural Information Processing Systems, 2019

562 citations recorded by the source corpus.

While we would like agents that can coordinate with humans, current algorithms such as self-play and population-based training create agents that can coordinate with themselves. Agents that assume their partner to be optimal or similar to them can converge to coordination protocols that fail to understand and be understood by humans. To demonstrate this, we introduce a simple environment that requires challenging coordination, based on the popular game Overcooked, and learn a simple model that mimics human play. We evaluate the performance of agents trained via self-play and population-based training. These agents perform very well when paired with themselves, but when paired with our human model, they are significantly worse than agents designed to play with the human model. An experiment with a planning algorithm yields the same conclusion, though only when the human-aware planner is given the exact human model that it is playing with. A user study with real humans shows this pattern as well, though less strongly. Qualitatively, we find that the gains come from having the agent adapt to the human's gameplay. Given this result, we suggest several approaches for designing agents that learn about humans in order to better coordinate with them. Code is available at https://github.com/HumanCompatibleAI/overcooked_ai.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/eb41a0cffa84d3d6035e6f5f420806ddc962b1e6

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Human-Timescale Adaptation in an Open-Ended Task Space (2023, methodology): “Inspired by the video game Overcooked (Carroll et al., 2019; Strouse et al., 2021), in this task both players need to “serve tomato soup to a hungry patron”.”
A Survey of Imitation Learning Methods, Environments and Metrics (2024, background): “One common approach among work such as [16, 18, 28, 120] is coupling behavioural cloning with other learning mechanisms, such as attention [28] or domain-shift [18, 120].”
Imitation Learning: A Survey of Learning Methods, Environments and Metrics (2024, unknown): “One common approach in these work [8, 56, 57, 64] is coupling behavioural cloning with other learning mechanisms, such as attention [57] or domain-shift [8, 56].”
Building Cooperative Embodied Agents Modularly with Large Language Models (2023, background): “However, these platforms either neglects communication (Jaderberg et al., 2019; Samvelyan et al., 2019; Carroll et al., 2019; Puig et al., 2021; 2023), or use uninterpretable continuous vectors (Jiang & Lu, 2018; Das et al., 2019) or limited discrete symbols (Lowe et al., 2017; Jaques et al., 2019;…”
Learning with Humans in the Loop (2020, background): “driving through an intersection) or there might be many equally good policies to complete a task and it is hard to know a priori which one a person will choose – approaches like self-play and population-based training are therefore insu cient when applied naively [12].”
Social Diversity and Social Preferences in Mixed-Motive Reinforcement Learning (2020, background): “Similarly, recent work on pure-common interest reinforcement learning in Markov games has highlighted the importance of robustness to diverse partner policies [1, 11].”
Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration (2020, background): “Platform Action Views Realistic Humanoid Human-like Agent Multi-agent Overcooked (Carroll et al., 2019) High/Low 3rd Person No No Yes Yes Malmo (Johnson et al.”
Imitating Human Behaviour with Diffusion Models (2023, background): “If we instead assume access to human behavioural data, collaborative agents can be improved by training with models of human behaviour (Carroll et al., 2019).”
ToM2C: Target-oriented Multi-agent Communication and Cooperation with Theory of Mind (2021, unknown): “(Puig et al., 2021; Carroll et al., 2019; Netanyahu et al., 2021) studies the human-AI cooperation with mental inference.”
"Other-Play" for Zero-Shot Coordination (2020, background): “A central challenge for AI is constructing agents that can coordinate and cooperate with partners they have not seen before (Kleiman-Weiner et al., 2016; Lerer & Peysakhovich, 2017; Carroll et al., 2019; Shum et al., 2019).”
Multi-agent Communication meets Natural Language: Synergies between Functional and Structural Language Learning (2020, background): “This pathological behaviour of self-play is not speciﬁc to language and extends to other policies (Carroll et al., 2019).”