Accelerating Online Reinforcement Learning with Offline Datasets

By Ashvin Nair, Murtaza Dalal, Abhishek Gupta, Sergey Levine. arXiv.org, 2020

858 citations recorded by the source corpus.

Reinforcement learning provides an appealing formalism for learning control policies from experience. However, the classic active formulation of reinforcement learning necessitates a lengthy active exploration process for each behavior, making it difficult to apply in real-world settings. If we can instead allow reinforcement learning to effectively use previously collected data to aid the online learning process, where the data could be expert demonstrations or more generally any prior experience, we could make reinforcement learning a substantially more practical tool. While a number of recent methods have sought to learn offline from previously collected data, it remains exceptionally difficult to train a policy with offline data and improve it further with online reinforcement learning. In this paper we systematically analyze why this problem is so challenging, and propose a novel algorithm that combines sample-efficient dynamic programming with maximum likelihood policy updates, providing a simple and effective framework that is able to leverage large amounts of offline data and then quickly perform online fine-tuning of reinforcement learning policies. We show that our method enables rapid learning of skills with a combination of prior demonstration data and online experience across a suite of difficult dexterous manipulation and benchmark tasks.

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning

View Paper: https://arxiv.org/abs/2006.09359

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

Pretraining Language Models with Human Preferences (2023, background): “Most work in offline RL has focused on pretraining policies for robotic control environments (Nair et al., 2020; Kumar et al., 2020).”
Offline Reinforcement Learning as One Big Sequence Modeling Problem (2021, background): “Modeling the states and actions jointly already provides a bias toward generating in-distribution actions, which avoids the need for explicit pessimism (Fujimoto et al., 2019; Kumar et al., 2019a; Ghasemipour et al., 2021; Nair et al., 2020; Jin et al., 2021; Yin et al., 2021; Dadashi et al., 2021).”
Conservative Q-Learning for Offline Reinforcement Learning (2020, background): “Most of these methods require a separately estimated model to the behavior policy, πβ(a|s) [15, 30, 59, 27, 54, 55], and are thus limited by their ability to accurately estimate the unknown behavior policy [42], which might be especially complex in settings where the data is collected from multiple ”
A Minimalist Approach to Offline Reinforcement Learning (2021, methodology): “Results for BRAC are obtained from the D4RL benchmark and from the CQL paper, while the AWAC results are obtained directly from the paper.”
EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL (2020, background): “Most prior ofﬂine RL works have not considered how their methods might transfer to online or batched online setting, and recent work (Nair et al., 2020) has demonstrated the challenges of ﬁnetuning from a policy trained ofﬂine, in the online setting.”
Critic Regularized Regression (2020, background): “Concurrently to our work, [25] proposed Advantage Weighted Actor Critic (AWAR) for accelerating online RL with offline datasets.”
Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization (2020, methodology): “…ofﬂine algorithms, namely, CQL (Kumar et al., 2020), BEAR (Kumar et al., 2019), BRAC (Wu et al., 2019), AWR (Peng et al., 2019) and BCQ (Fujimoto et al., 2019), on the D4RL MuJoCo locomotion datasets (Fu et al., 2020), used as standard ofﬂine RL benchmarks (Kumar et al., 2020; Nair et al., 2020).”
Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning (2022, methodology): “…how far the policy can deviate from the behavior policy (Fujimoto et al., 2019; Fujimoto & Gu, 2021; Kumar et al., 2019; Wu et al., 2019; Nair et al., 2020; Lyu et al., 2022); 2) constraining the learned value function to assign low values to out-of-distribution actions (Kostrikov et…”
Efficient Online Reinforcement Learning with Offline Data (2023, methodology): “Notably, we outperform the best reported performance on the Sparse Adroit ‘Door’ task by 2.5×.”
Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning (2021, methodology): “There are also some previous works using softmax to deal with the partition function, such as AWAC [29]) and VMPO [45], which has been conﬁrmed to promote performance improvement.”
A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems (2022, methodology): “Interestingly, Jiang and Li [39] do not attempt to correct this bias, likely because their advantage estimate would still be biased due to its reliance on a bootstrapped return.”
OpenChat: Advancing Open-source Language Models with Mixed-Quality Data (2023, methodology): “We can thus extract the optimized policy πθ by minimizing the KL divergence between π∗ under the class-conditioned dataset Dc (Nair et al., 2020; Korbak et al., 2022):”
Is Pessimism Provably Efficient for Offline RL? (2020, background): “…Wu et al., 2019; Kumar et al., 2019; 2020; Agarwal et al., 2020b; Yu et al., 2020; Kidambi et al., 2020; Wang et al., 2020c; Siegel et al., 2020; Nair et al., 2020; Liu et al., 2020) demonstrate the empirical success of various algorithms, which fall into two (possibly overlapping) categories:…”
Planning with Diffusion for Flexible Behavior Synthesis (2022, background): “, 2022), and AWAC (Nair et al., 2020); return-conditioning approaches like Decision Transformer (DT; Chen et al.”