Simplified Action Decoder for Deep Multi-Agent Reinforcement Learning

By Hengyuan Hu, Jakob N. Foerster. 2019

89 citations recorded by the source corpus.

In recent years we have seen fast progress on a number of benchmark problems in AI, with modern methods achieving near or super human performance in Go, Poker and Dota. One common aspect of all of these challenges is that they are by design adversarial or, technically speaking, zero-sum. In contrast to these settings, success in the real world commonly requires humans to collaborate and communicate with others, in settings that are, at least partially, cooperative. In the last year, the card game Hanabi has been established as a new benchmark environment for AI to fill this gap. In particular, Hanabi is interesting to humans since it is entirely focused on theory of mind, i.e., the ability to effectively reason over the intentions, beliefs and point of view of other agents when observing their actions. Learning to be informative when observed by others is an interesting challenge for Reinforcement Learning (RL): Fundamentally, RL requires agents to explore in order to discover good policies. However, when done naively, this randomness will inherently make their actions less informative to others during training. We present a new deep multi-agent RL method, the Simplified Action Decoder (SAD), which resolves this contradiction exploiting the centralized training phase. During training SAD allows other agents to not only observe the (exploratory) action chosen, but agents instead also observe the greedy action of their team mates. By combining this simple intuition with best practices for multi-agent learning, SAD establishes a new SOTA for learning methods for 2-5 players on the self-play part of the Hanabi challenge. Our ablations show the contributions of SAD compared with the best practice components. All of our code and trained agents are available at this https URL.

Subfields: Artificial Intelligence, Economics and Econometrics, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Adam Lerer
2. Jakob N. Foerster
3. Hengyuan Hu
4. Alex Peysakhovich
5. Noam Brown
6. Yi Wu
7. Yu Wang
8. Sarath Chandar

In Economics and Econometrics

1. Adam Lerer
2. Jakob N. Foerster
3. Hengyuan Hu
4. Noam Brown
5. Yi Wu
6. Yu Wang
7. Chao Yu
8. Sarath Chandar

In Reinforcement Learning

1. Adam Lerer
2. Hengyuan Hu
3. Jakob N. Foerster
4. Noam Brown
5. Alex Peysakhovich
6. Yi Wu
7. Yu Wang
8. Chao Yu

Citations in the corpus

• The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games (2021, methodology): “Experiment Setup: We compare MAPPO and IPPO to off-policy baselines including MADDPG [21] in the MPEs; QMix [25] and SOTA methods including RODE [36], QPlex [35] in SMAC; SAD [15] in Hanabi; and QMix and SOTA methods including CDS [19] and TiKick [16] in GRF.”
• Continuous Coordination As a Realistic Scenario for Lifelong Learning (2021, methodology): “In recent years, there has been rapid progress on novel MARL algorithms that are based on centralized training with decentralized execution (Self-Play) as a training paradigm (Sunehag et al., 2017; Foerster et al., 2017; Hong et al., 2017; Foerster et al., 2019; Hu & Foerster, 2019).”
• Learned Belief Search: Efficiently Improving Policies in Partially Observable Settings (2021, methodology): “The state-of-the-art in Hanabi self-play is achieved by the SPARTA Monte Carlo search algorithm applied to the SAD blueprint policy (Lerer et al., 2020).”
• "Other-Play" for Zero-Shot Coordination (2020, methodology): “In Hanabi the Simplified Action Decoder (SAD) (Hu & Foerster, 2019) method achieves SOTA performance for RL agents.”
• The Surprising Effectiveness of MAPPO in Cooperative, Multi-Agent Games (2021, methodology): “Note that SAD uses about 10B timesteps 2 and reports the best and average performance over 13 trials.”
• Improving Policies via Search in Cooperative Partially Observable Games (2019, methodology): “The Simplified Action Decoder algorithm established the most recent state of the art in Hanabi (Hu and Foerster 2020).”