Regularization Matters in Policy Optimization
By Zhuang Liu, Xuanlin Li, Bingyi Kang, Trevor Darrell. 2019
39 citations recorded by the source corpus.
Deep Reinforcement Learning (Deep RL) has been receiving increasingly more attention thanks to its encouraging performance on a variety of control tasks. Yet, conventional regularization techniques in training neural networks (e.g., $L_2$ regularization, dropout) have been largely ignored in RL methods, possibly because agents are typically trained and evaluated in the same environment. In this work, we present the first comprehensive study of regularization techniques with multiple policy optimization algorithms on continuous control tasks. Interestingly, we find conventional regularization techniques on the policy networks can often bring large improvement on the task performance, and the improvement is typically more significant when the task is more difficult. We also compare with the widely used entropy regularization and find $L_2$ regularization is generally better. Our findings are further confirmed to be robust against the choice of training hyperparameters. We also study the effects of regularizing different components and find that only regularizing the policy network is typically enough. We hope our study provides guidance for future practices in regularizing policy optimization algorithms.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Shimon Whiteson
- Pieter Abbeel
- Rob Fergus
- Ilya Kostrikov
- Denis Yarats
- Gregory Farquhar
- Chelsea Finn
In Computer Vision and Pattern Recognition
- Ilya Kostrikov
- Denis Yarats
- Shimon Whiteson
- Rob Fergus
- Abhinav Kumar Gupta
- Luc Van Gool
- Gregory Farquhar
- Trevor Darrell
Citations in the corpus
- Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels (2020, background): “See (Liu et al., 2019) for a comprehensive evaluation of different network regularization methods.”
- Transient Non-stationarity and Generalisation in Deep Reinforcement Learning (2020, background): “, 2017), or regularise the network (Cobbe et al., 2019b; Igl et al., 2019; Liu et al., 2019).”