Critic Regularized Regression

By Ziyun Wang, Alexander Novikov, Konrad Zolna, Jost Tobias Springenberg, Scott E. Reed, Bobak Shahriari, Noah Siegel, Josh Merel, Caglar Gulcehre, Nicolas Manfred Otto Heess and 1 more. 2020

389 citations recorded by the source corpus.

Offline reinforcement learning (RL), also known as batch RL, offers the prospect of policy optimization from large pre-recorded datasets without online environment interaction. It addresses challenges with regard to the cost of data collection and safety, both of which are particularly pertinent to real-world applications of RL. Unfortunately, most off-policy algorithms perform poorly when learning from a fixed dataset. In this paper, we propose a novel offline RL algorithm to learn policies from data using a form of critic-regularized regression (CRR). We find that CRR performs surprisingly well and scales to tasks with high-dimensional state and action spaces -- outperforming several state-of-the-art offline RL algorithms by a significant margin on a wide range of benchmark tasks.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Reinforcement Learning, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Reinforcement Learning

In Optimization & Training

Citations in the corpus

Offline Reinforcement Learning with Implicit Q-Learning (2021, methodology): “…via an explicit density model (Wu et al., 2019; Fujimoto et al., 2019; Kumar et al., 2019), implicit divergence constraints (Nair et al., 2020; Wang et al., 2020; Peters & Schaal, 2007; Peng et al., 2019), or by adding a supervised learning term to the policy improvement objective (Fujimoto &…”
Offline RL Without Off-Policy Evaluation (2021, methodology): “Most algorithms from the literature fall into this category [Fujimoto et al., 2018a, Kumar et al., 2019, 2020, Wu et al., 2019, Wang et al., 2020b, Siegel et al., 2020].”
Continuous Doubly Constrained Batch Reinforcement Learning (2021, methodology): “Although methods like [46, 60] do not have an explicit constraint on the policy update, they still can be categorized as a policy constrained-based approach as the policy update rule has been changed in a such a way that it selectively updates the policy utilizing information contained in the Q-valu”
Benchmarks for Deep Off-Policy Evaluation (2021, methodology): “Datasets and policies We train four offline RL algorithms (D4PG (Barth-Maron et al., 2018), ABM (Siegel et al., 2020), CRR (Wang et al., 2020) and behavior cloning), varying their hyperparameters.”
Hyperparameter Selection for Offline Reinforcement Learning (2020, methodology): “In this paper, we use Critic Regularized Regression (CRR; Wang et al. (2020)) as our main oﬄine RL method; CRR uses the advantage function to select the best actions in the dataset for training behavior cloning.”
Acme: A Research Framework for Distributed Reinforcement Learning (2020, background): “CRR Continuous Actions Offline Q-Network Policy-Network Bootstrapping Critic-Regularized Regression (CRR) (Wang et al., 2020) is a modification of behavior cloning that restricts training to those example transitions whose advantage (i.”
A Generalist Agent (2022, methodology): “Figure 10 compares the success rate of Gato across different fine-tuning data regimes to the sim-to-real expert and a Critic-Regularized Regression (CRR) (Wang et al., 2020) agent trained on 35k episodes of all test triplets.”
NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning (2021, methodology): “Meanwhile, offline RL methods are often pessimistic about the out-of-data distribution, by constraining the RL agent to be close to the offline data [12, 17], or only trusting the learned environment model when the uncertainty of the generated data is low [27, 28].”
Challenges of real-world reinforcement learning: definitions, benchmarks and analysis (2021, methodology): “15 Learning from offline data on large datasets in the easy combined challenge setting using CRR on quadruped Fig.”
A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems (2022, methodology): “Furthermore, Wang et al. [45] propose another method named critic regularized regression (CRR), which also uses the indicator function to aggressively filter out below-average actions like in (47), but opts for a more pessimistic advantage estimator where we change the expectation to a max operator ”
The Importance of Pessimism in Fixed-Dataset Policy Optimization (2020, background): “Recently, several deep-learningbased algorithms for xed-dataset policy optimization have been proposed [7, 14, 17, 18, 19, 21, 43, 46, 48].”
Model-Based Offline Planning (2020, methodology): “Learning from logs of a system is often called ‘Offline Reinforcement Learning’ (Wu et al., 2019; Peng et al., 2019; Fujimoto et al., 2019; Wang et al., 2020) and both model-free (Wu et al.”
Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning (2021, methodology): “[66] consider an indicator function f : x ∈ R 7→ 1[x ≥ 0], whereas Oh et al.”
Batch Reinforcement Learning Through Continuation Method (2021, methodology): “Recent works [29, 33] studied policy iteration under batch RL.”
IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies (2023, neutral): “To this end, recent methods use implicit divergence constraints [38, 37, 33, 45], explicit density models [47, 13, 27, 15], and supervised learning terms [12].”
Is Pessimism Provably Efficient for Offline RL? (2020, background): “…Laroche et al., 2019; Jaques et al., 2019; Wu et al., 2019; Kumar et al., 2019; 2020; Agarwal et al., 2020b; Yu et al., 2020; Kidambi et al., 2020; Wang et al., 2020c; Siegel et al., 2020; Nair et al., 2020; Liu et al., 2020) demonstrate the empirical success of various algorithms, which fall…”