Conservative Q-Learning for Offline Reinforcement Learning
By Aviral Kumar, Aurick Zhou, George Tucker, Sergey Levine. Neural Information Processing Systems, 2020
2,669 citations recorded by the source corpus.
Effectively leveraging large, previously collected datasets in reinforcement learning (RL) is a key challenge for large-scale real-world applications. Offline RL algorithms promise to learn effective policies from previously-collected, static datasets without further interaction. However, in practice, offline RL presents a major challenge, and standard off-policy RL methods can fail due to overestimation of values induced by the distributional shift between the dataset and the learned policy, especially when training on complex and multi-modal data distributions. In this paper, we propose conservative Q-learning (CQL), which aims to address these limitations by learning a conservative Q-function such that the expected value of a policy under this Q-function lower-bounds its true value. We theoretically show that CQL produces a lower bound on the value of the current policy and that it can be incorporated into a policy learning procedure with theoretical improvement guarantees. In practice, CQL augments the standard Bellman error objective with a simple Q-value regularizer which is straightforward to implement on top of existing deep Q-learning and actor-critic implementations. On both discrete and continuous control domains, we show that CQL substantially outperforms existing offline RL methods, often learning policies that attain 2-5 times higher final return, especially when learning from complex and multi-modal data distributions.
Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning
View Paper: https://arxiv.org/abs/2006.04779
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Pieter Abbeel
- Yoshua Bengio
- Chelsea Finn
- Ruslan Salakhutdinov
- Mohit Bansal
- Quoc V. Le
- Trevor Darrell
In Management Science and Operations Research
- Sergey Levine
- Ofir Nachum
- Aviral Kumar
- Shixiang Shane Gu
- George Tucker
- Pieter Abbeel
- Chelsea Finn
- Nan Jiang
In Reinforcement Learning
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Ofir Nachum
- Shixiang Shane Gu
- Aviral Kumar
- Ilya Kostrikov
- George Tucker
Citations in the corpus
- Pretraining Language Models with Human Preferences (2023, background): “Most work in offline RL has focused on pretraining policies for robotic control environments (Nair et al., 2020; Kumar et al., 2020).”
- Decision Transformer: Reinforcement Learning via Sequence Modeling (2021, methodology): “We show full results comparing %BC to Decision Transformer and CQL in Table 3, sweeping over X ∈ [10% , 25% , 40% , 100”
- Offline Reinforcement Learning as One Big Sequence Modeling Problem (2021, methodology): “…works have studied applying sequence models of various types to represent components in standard RL algorithms, such as policies, value functions, and models (Bakker, 2002; Heess et al., 2015a; Chiappa et al., 2017; Parisotto et al., 2020; Parisotto & Salakhutdinov, 2021; Kumar et al., 2020b).”
- Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “A variant of this issue was noted in the standard RL setting, referred to as an absence of “corrective feedback” by Kumar et al. (2020) (see Kumar and Gupta (2020) for a short summary), and such a problem may affect offline RL algorithms with function approximation more severely, since they have no…”
- Offline Reinforcement Learning with Implicit Q-Learning (2021, methodology): “50 E p i s d e e t u In the former category, we compare to CQL (Kumar et al., 2020), TD3+BC (Fu-jimoto & Gu, 2021), and AWAC (Nair et al., 2020). n We compare to methods that are representative of both multi-step dynamic programming and one-step approaches.”
- What Matters in Learning from Offline Human Demonstrations for Robot Manipulation (2021, methodology): “We chose to evaluate 6 algorithms in this study – Behavioral Cloning (BC), BC with an RNN policy (BC-RNN), Hierarchical Behavioral Cloning (HBC) [10], Batch-Constrained Q-Learning (BCQ) [26], Conservative Q-Learning (CQL) [27], and IRIS [20].”
- Implicit Behavioral Cloning (2021, methodology): “Baselines Ours Explicit Implicit Explicit Implicit Method NearestBC CQL [26] S4RL [27] BC (MSE) BC (EBM) BC (MSE) BC (EBM) Neighbor (from CQL [26]) w/ RWR [28] w/ RWR [28] Uses data (o,a) (o,a) (o,a,r) (o,a,r) (o,a) (o,a) (o,a,r) (o,a,r) Domain Task Name”
- Offline RL Without Off-Policy Evaluation (2021, methodology): “…the learned policy from choosing unknown actions is to incorporate some form of regularization to encourage staying near the behavior and being pessimistic about unknown state, action pairs [Wu et al., 2019, Nachum et al., 2019, Kumar et al., 2020, Kostrikov et al., 2021, Gulcehre et al., 2021].”
- A Minimalist Approach to Offline Reinforcement Learning (2021, methodology): “Our offline RL baselines include two state-of-the-art algorithms, CQL [Kumar et al., 2020] and FisherBRC [Kostrikov et al., 2021], as well as BRAC [Wu et al., 2019] and AWAC [Nair et al., 2020] due to their algorithmic simplicity.”
- Regularized Behavior Value Estimation (2021, methodology): “…et al., 2019b; Kumar et al., 2019; Jaques et al., 2019; Siegel et al., 2020; Wang et al., 2020; Ghasemipour et al., 2020), 2) value-based approaches lower value estimates for unseen state-actions pairs, either through regularization or uncertainty (Kumar et al., 2020; Agarwal et al., 2019a).”
- Offline Reinforcement Learning with Fisher Divergence Critic Regularization (2021, methodology): “Connections to CQL The CQL objective, although originally derived in Kumar et al. (2020) from a very different perspective, can also be motivated as a regularizer on a Boltzmann policy.”
- Continuous Doubly Constrained Batch Reinforcement Learning (2021, methodology): “The same procedure is also used in previous works (Kumar et al., 2020; Fu et al., 2020) to report results and compare various batch RL methods.”
- EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL (2020, methodology): “Additionally, replacing the Q function ensembles in EMaQ with conservative estimates through CQL may be a fruitful avenue for future work.”
- COMBO: Conservative Offline Model-Based Policy Optimization (2021, methodology): “We achieve this by extending conservative Q-learning [29], which does not require explicit uncertainty quantification, into the model-based setting.”
- Representation Matters: Offline Pretraining for Sequential Decision Making (2021, background): “Accordingly, much of the recent offline RL research focuses on proposing new policy optimization algorithms amenable to learning from offline datasets (e.g., Fujimoto et al. (2019); Wu et al. (2019); Agarwal et al. (2020); Kumar et al. (2020); Yu et al. (2020); Matsushima et al. (2020)).”
- OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning (2020, methodology): “We used CQL (Kumar et al., 2020b) for offline RL experiments in learning a task policy either in action space A or latent space Z .”
- TiKick: Towards Playing Multi-agent Football Full Games from Single-agent Demonstrations (2021, neutral): “CQL : CQL (Kumar et al., 2020) is an offline reinforcement learning algorithm that tries to learn conservative Q-values via adding penalties on the Q-functions.”
- Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization (2020, methodology): “More recently, a related but distinct problem known as offline RL has gained popularity (Levine et al., 2020; Wu et al., 2019; Agarwal et al., 2019; Kumar et al., 2020).”
- Can Wikipedia Help Offline Reinforcement Learning? (2022, methodology): “In addition to benchmarking our pre-trained language models, we compare to popular state-of-the-art of-fline RL algorithms as follows: Decision Transformer (DT) (Chen et al., 2021), CQL (Kumar et al., 2020), TD3+BC (Fujimoto & Gu, 2021), BRAC (Wu et al., 2019), and AWR baselines (Peng et al., 2019).”