Consistent Dropout for Policy Gradient Reinforcement Learning
By Matthew J. Hausknecht, Nolan Wagener. 2022
14 citations recorded by the source corpus.
Dropout has long been a staple of supervised learning, but is rarely used in reinforcement learning. We analyze why naive application of dropout is problematic for policy-gradient learning algorithms and introduce consistent dropout, a simple technique to address this instability. We demonstrate consistent dropout enables stable training with A2C and PPO in both continuous and discrete action environments across a wide range of dropout probabilities. Finally, we show that consistent dropout enables the online training of complex architectures such as GPT without needing to disable the model's native dropout.
Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning, Optimization & Training
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Hannaneh Hajishirzi
- Yejin Choi
- Jack Hessel
- Prithviraj Ammanabrolu
- Mohit Bansal
- Luke S. Zettlemoyer
- Rafet Sifa
- Christian Bauckhage
In Management Science and Operations Research
In Reinforcement Learning
- Hannaneh Hajishirzi
- Yejin Choi
- Kianté Brantley
- Christian Bauckhage
- Prithviraj Ammanabrolu
- Rafet Sifa
- Rajkumar Ramamurthy
- Jack Hessel
In Optimization & Training
- Kianté Brantley
- Christian Bauckhage
- Yejin Choi
- Prithviraj Ammanabrolu
- Rafet Sifa
- Hannaneh Hajishirzi
- Jack Hessel
Citations in the corpus
- Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization (2022, methodology): “The first is dropout, which in its standard form was found to cause instability in policy gradient methods in continuous control settings by Hausknecht & Wagener (2022).”