Mastering Complex Control in MOBA Games with Deep Reinforcement Learning

By Deheng Ye, Zhao Liu, Mingfei Sun, Bei Shi, Peilin Zhao, Hao Wu, Hongsheng Yu, Shaojie Yang, Xipeng Wu, Qingwei Guo and 8 more. AAAI Conference on Artificial Intelligence, 2019

410 citations recorded by the source corpus.

We study the reinforcement learning problem of complex action control in the Multi-player Online Battle Arena (MOBA) 1v1 games. This problem involves far more complicated state and action spaces than those of traditional 1v1 games, such as Go and Atari series, which makes it very difficult to search any policies with human-level performance. In this paper, we present a deep reinforcement learning framework to tackle this problem from the perspectives of both system and algorithm. Our system is of low coupling and high scalability, which enables efficient explorations at large scale. Our algorithm includes several novel strategies, including control dependency decoupling, action mask, target attention, and dual-clip PPO, with which our proposed actor-critic network can be effectively trained in our system. Tested on the MOBA game Honor of Kings, the trained AI agents can defeat top professional human players in full 1v1 games.

Subfields: Artificial Intelligence, Sociology and Political Science

View Paper: https://www.semanticscholar.org/paper/292bbd1287a0674cd9e3e79224e768ca557dcf81

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Joseph J. Lim
2. Shengyi Huang
3. Peilin Zhao
4. Gao Huang
5. Deheng Ye
6. Norio Kosaka
7. KyungHyun Kim
8. Sheng Chen

In Sociology and Political Science

1. Gao Huang
2. Peilin Zhao
3. Joseph J. Lim

Citations in the corpus

• Supervised Learning Achieves Human-Level Performance in MOBA Games: A Case Study of Honor of Kings (2020, methodology): “For RL, a typical way of generating training data is through self-play [3], [4], which requires determining a pool of heroes before training.”
• Know Your Action Set: Learning Action Relations for Reinforcement Learning (2022, background): “An instance of SAS-MDPs is when certain actions become invalid, like in games, they are masked out from the output action probability distribution (Huang & Ontañón, 2020; Ye et al., 2020; Kanervisto et al., 2020).”
• Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning (2021, background): “Recently, reinforcement learning (RL), an active learning process, has achieved massive success in various domains ranging from strategy games [59] to recommendation systems [8].”
• A Closer Look at Invalid Action Masking in Policy Gradient Algorithms (2020, positive): “Lastly, Kanervisto, et al. (Kanervisto, Scheller, and Hau-tam¨aki 2020) and Ye, et al. (Ye et al. 2020) provide ablation studies to show invalid action masking could be important to the performance of agents, but they do not study the empirical effect of invalid action masking as the space of invali”