Off-Policy Deep Reinforcement Learning without Exploration

By Scott Fujimoto, David Meger, Doina Precup. 2018

2,096 citations recorded by the source corpus.

Many practical applications of reinforcement learning constrain agents to learn from a fixed batch of data which has already been gathered, without offering further possibility for data collection. In this paper, we demonstrate that due to errors introduced by extrapolation, standard off-policy deep reinforcement learning algorithms, such as DQN and DDPG, are incapable of learning with data uncorrelated to the distribution under the current policy, making them ineffective for this fixed batch setting. We introduce a novel class of off-policy algorithms, batch-constrained reinforcement learning, which restricts the action space in order to force the agent towards behaving close to on-policy with respect to a subset of the given data. We present the first continuous control deep reinforcement learning algorithm which can learn effectively from arbitrary, fixed batch data, and empirically demonstrate the quality of its behavior in several tasks.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog (2019, methodology): “Note that since we operate in discrete action space, we could not include the perturbation model originally proposed by [16], which may be critical to achieving good performance with BCQ.”
Human-centric Dialog Training via Offline Reinforcement Learning (2020, methodology): “Most similar to our work is Batch Constrained Q-learning (BCQ) (Fujimoto et al., 2018), which addresses extrapolation error in offline RL by constraining the actions of the policy to be close to the offline data.”
Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning (2019, methodology): “as TD3 and SAC, can in principle handle off-policy data but, as discussed in prior work, tend to struggle in this setting in practice (Fujimoto et al., 2019; Kumar et al., 2019).”
Decision Transformer: Reinforcement Learning via Sequence Modeling (2021, methodology): “To mitigate the impact of distribution shift in offline RL, prior algorithms either (a) constrain the policy action space (Fujimoto et al., 2019; Kumar et al., 2019a; Siegel et al., 2020) or (b) incorporate value pessimism (Fujimoto et al.”
Offline Reinforcement Learning as One Big Sequence Modeling Problem (2021, methodology): “distribution behavior that is not supported under the dataset (Fujimoto et al., 2019; Kumar et al., 2019a; Ghasemipour et al., 2021); imitation learning, where it is used to fit an expert’s actions to obtain a policy (Ross & Bagnell, 2010; Ross et al.”
Toward Causal Representation Learning (2021, background): “Off-line RL [73], [150] tries to address this concern by learning a policy from a fixed data set of trajectories, without requiring any experimental or interventional data (i.”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “…in recent work is to utilize training data collected via a standard online reinforcement learning algorithm, using either the entire replay buffer for an off-policy algorithm for training (Kumar et al., 2019a; Agarwal et al., 2019; Fujimoto et al., 2018), or even data from the optimal policy.”
Conservative Q-Learning for Offline Reinforcement Learning (2020, methodology): “Most of these methods require a separately estimated model to the behavior policy, πβ(a|s) [15, 30, 59, 27, 54, 55], and are thus limited by their ability to accurately estimate the unknown behavior policy [42], which might be especially complex in settings where the data is collected from multiple ”
Offline Reinforcement Learning with Implicit Q-Learning (2021, methodology): “Prior methods generally address this by either constraining the policy to limit how far it deviates from the behavior policy (Fujimoto et al., 2019; Wu et al., 2019; Fujimoto & Gu, 2021; Kumar et al., 2019; Nair et al., 2020; Wang et al., 2020), or by regularizing the learned value functions to…”
Accelerating Online Reinforcement Learning with Offline Datasets (2020, methodology): “While prior methods for fully offline RL provide a mechanism for utilizing offline data [5, 9], as we will show in our experiments, such methods generally are not effective for fine-tuning with online data as they are often too conservative.”
Towards Causal Representation Learning (2021, background): “Offline RL [72, 150] tries to address this concern by learning a policy from a fixed dataset of trajectories, without requiring any experimental or interventional data (i.”
Sample-Efficient Reinforcement Learning via Counterfactual-Based Data Augmentation (2020, methodology): “The three SOTA approaches are STEVE [6], BCQ [29], and PETS [18].”
Adaptive Online Planning for Continual Lifelong Learning (2019, background): “Note that since the AOP policy is learned from off-policy data (the planner), it suffers from divergence issues and should be weaker than TD3 on its own [11].”
Off-policy Bandits with Deficient Support (2020, background): “It emphasizes that function approximation equipped with Q-learning can diverge in the off-policy learning setting, making most off-policy RL methods very conservative in extrapolation because of the severe error-propagation issue [7, 16].”
MOPO: Model-based Offline Policy Optimization (2020, methodology): “To do so, a wide range of ofﬂine RL methods propose to mitigate bootstrapped error by constraining the learned policy to the behavior policy induced by the dataset [18, 35, 70, 29, 48, 51, 57].”
MOReL : Model-Based Offline Reinforcement Learning (2020, methodology): “This is similar to evaluation protocols followed in prior work [18, 15, 16].”
D4RL: Datasets for Deep Data-Driven Reinforcement Learning (2020, methodology): “This evaluation protocol has been used in domains such as continuous control for robotics [Fujimoto et al., 2018, Kumar et al., 2019, Wu et al., 2019], navigation [Laroche et al., 2019], industrial control [Hein et al., 2017], and Atari video games [Agarwal et al., 2019b].”
DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction (2020, background): “Our work bears some similarity to the recent body of literature on batch, or offline reinforcement learning (Kumar et al., 2019; Fujimoto et al., 2019; Wu et al., 2019), where the goal is to learn an effective policy, using access to only a finite, off-policy dataset collected previously.”
Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning (2020, methodology): “In contrast the other methods learn stably in the high-data regime, with BCQ lagging behind in Hopper and Quadruped (sticking too close to the VAE prior actions, an effect already observed in (Kumar et al., 2019)).”