Regularized Behavior Value Estimation
By Caglar Gulcehre, Sergio Gomez Colmenarejo, Ziyun Wang, Jakub Sygnowski, T. Paine, Konrad Zolna, Yutian Chen, Matthew W. Hoffman, Razvan Pascanu, Nando de Freitas. 2021
43 citations recorded by the source corpus.
Offline reinforcement learning restricts the learning process to rely only on logged-data without access to an environment. While this enables real-world applications, it also poses unique challenges. One important challenge is dealing with errors caused by the overestimation of values for state-action pairs not well-covered by the training data. Due to bootstrapping, these errors get amplified during training and can lead to divergence, thereby crippling learning. To overcome this challenge, we introduce Regularized Behavior Value Estimation (R-BVE). Unlike most approaches, which use policy improvement during training, R-BVE estimates the value of the behavior policy during training and only performs policy improvement at deployment time. Further, R-BVE uses a ranking regularisation term that favours actions in the dataset that lead to successful outcomes. We provide ample empirical evidence of R-BVE's effectiveness, including state-of-the-art performance on the RL Unplugged ATARI dataset. We also test R-BVE on new datasets, from bsuite and a challenging DeepMind Lab task, and show that R-BVE outperforms other state-of-the-art discrete control offline RL methods.
Subfields: Artificial Intelligence, Reinforcement Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Nando de Freitas
- Pieter Abbeel
- Yoshua Bengio
- Quoc V. Le
- Ruslan Salakhutdinov
- Kyunghyun Cho
- Ziyun Wang
In Reinforcement Learning
- Sergey Levine
- Ziyun Wang
- T. Paine
- Nando de Freitas
- Alexander Novikov
- Sergio Gomez Colmenarejo
- Pieter Abbeel
- Gabriel Barth-Maron
Citations in the corpus
- Offline RL Without Off-Policy Evaluation (2021, methodology): “…the learned policy from choosing unknown actions is to incorporate some form of regularization to encourage staying near the behavior and being pessimistic about unknown state, action pairs [Wu et al., 2019, Nachum et al., 2019, Kumar et al., 2020, Kostrikov et al., 2021, Gulcehre et al., 2021].”
- Acme: A Research Framework for Distributed Reinforcement Learning (2020, background): “BVE Discrete Actions Offline Q-Network Bootstrapping Behavior value estimation (BVE) (Gulcehre et al., 2021) is a one-step policy improvement algorithm where the agent, instead of trying to recover the optimal policy, learns a Q-function that estimates the behavior value Q (s, a) from the dataset.”
- Reinforced Self-Training (ReST) for Language Modeling (2023, methodology): “However, the quality of the policy learnt offline inevitably depends on the properties of the offline dataset (Fu et al., 2020; Gulcehre et al., 2021).”