A Closer Look at Invalid Action Masking in Policy Gradient Algorithms
By Shengyi Huang, Santiago Ontan'on. 2020
526 citations recorded by the source corpus.
In recent years, Deep Reinforcement Learning (DRL) algorithms have achieved state-of-the-art performance in many challenging strategy games. Because these games have complicated rules, an action sampled from the full discrete action distribution predicted by the learned policy is likely to be invalid according to the game rules (e.g., walking into a wall). The usual approach to deal with this problem in policy gradient algorithms is to “mask out” invalid actions and just sample from the set of valid actions. The implications of this process, however, remain under-investigated. In this paper, we 1) show theoretical justification for such a practice, 2) empirically demonstrate its importance as the space of invalid actions grows, and 3) provide further insights by evaluating different action masking regimes, such as removing masking after an agent has been trained using masking.
Subfields: Artificial Intelligence, Reinforcement Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Hannaneh Hajishirzi
- Yejin Choi
- Jack Hessel
- Prithviraj Ammanabrolu
- Mohit Bansal
- Luke S. Zettlemoyer
- Joseph J. Lim
- Rafet Sifa
In Reinforcement Learning
- Hannaneh Hajishirzi
- Yejin Choi
- Rousslan Fernand Julien Dossa
- Jeff Braga
- Chang Ye
- Kianté Brantley
- Christian Bauckhage
- Prithviraj Ammanabrolu
Citations in the corpus
- Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization (2022, methodology): “To address this issue, we introduce NLPO (Natural Language Policy Optimization), which is inspired by work on action elimination/invalid-action masking (Zahavy et al., 2018; Huang & Ontañón, 2020; Ammanabrolu & Hausknecht, 2020).”
- Know Your Action Set: Learning Action Relations for Reinforcement Learning (2022, background): “An instance of SAS-MDPs is when certain actions become invalid, like in games, they are masked out from the output action probability distribution (Huang & Ontañón, 2020; Ye et al., 2020; Kanervisto et al., 2020).”
- CleanRL: High-quality Single-file Implementations of Deep Reinforcement Learning Algorithms (2022, methodology): “Being able to display the variant’s differences explicitly has helped us explain 37 implementation details of PPO (Huang et al., 2022).”