"Other-Play" for Zero-Shot Coordination

By Hengyuan Hu, Adam Lerer, Alex Peysakhovich, Jakob N. Foerster. 2020

287 citations recorded by the source corpus.

We consider the problem of zero-shot coordination - constructing AI agents that can coordinate with novel partners they have not seen before (e.g. humans). Standard Multi-Agent Reinforcement Learning (MARL) methods typically focus on the self-play (SP) setting where agents construct strategies by playing the game with themselves repeatedly. Unfortunately, applying SP naively to the zero-shot coordination problem can produce agents that establish highly specialized conventions that do not carry over to novel partners they have not been trained with. We introduce a novel learning algorithm called other-play (OP), that enhances self-play by looking for more robust strategies, exploiting the presence of known symmetries in the underlying problem. We characterize OP theoretically as well as experimentally. We study the cooperative card game Hanabi and show that OP agents achieve higher scores when paired with independently trained agents. In preliminary results we also show that our OP agents obtains higher average scores when paired with human players, compared to state-of-the-art SP agents.

Subfields: Artificial Intelligence, Economics and Econometrics, Reinforcement Learning, Game & Decision Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Economics and Econometrics

In Reinforcement Learning

In Game & Decision Theory

Citations in the corpus

Continuous Coordination As a Realistic Scenario for Lifelong Learning (2021, methodology): “The closest work to ours from the perspective of zero-shot coordination is Other-Play (Hu et al., 2020), which exploits the symmetry in the environment by training a self-play agent with shuffled observation space.”
Learned Belief Search: Efficiently Improving Policies in Partially Observable Settings (2021, methodology): “The state-of-the-art in Hanabi self-play is achieved by the SPARTA Monte Carlo search algorithm applied to the SAD blueprint policy (Lerer et al., 2020).”
Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomization (2021, methodology): “, 2020), which is a special case of ours, or learning robust policies (Li et al., 2019; Shen & How, 2019; Hu et al., 2020), which complements our method.”
SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents (2023, background): “Table 2 presents the models’ average scores when interacting with different partner models (i.e., the model it is paired with in interaction, Fu et al. 2023; Hu et al. 2020).”
Few-shot Language Coordination by Modeling Theory of Mind (2021, background): “Hu et al. (2020) propose other-play which maximizes the expected reward working with random partners.”
AI-QMIX: Attention and Imagination for Dynamic Multi-Agent Reinforcement Learning (2020, background): “While existing lines of research have explored curriculum learning to scale from small to larger scenarios [1, 21, 36] or collaboration with agents not seen during training [31, 4, 14], to our knowledge, no other works have explored the setting of simultaneous learning across diverse scenarios.”
Human-Timescale Adaptation in an Open-Ended Task Space (2023, background): “When faced with a new problem, agents must adapt on-the-ﬂy to agree on a single equilibrium of mutual beneﬁt (Stone et al., 2010; Hu et al., 2020; Christianos et al., 2022).”
Off-Belief Learning (2021, methodology): “For reference, we can also see from the same Figure 4 that the Other-Play Rank Bot and Color Bot have strong preferences to use only rank and only color to exchange information respectively, making it difﬁcult for them to coordinate with each other.”