An Optimistic Perspective on Offline Reinforcement Learning

By Rishabh Agarwal, Dale Schuurmans, Mohammad Norouzi. 2019

581 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Decision Transformer: Reinforcement Learning via Sequence Modeling (2021, methodology): “For REM and QR-DQN, there is a slight discrepancy between Agarwal et al. [13] and Kumar et al. [14]; we report raw data provided to us by REM authors.”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “Some prior works on offline RL have also noted that, for some types of datasets, conventional dynamic programming algorithms, such as deep Q-learning or deterministic actor-critic algorithms, can in fact work reasonably well (Agarwal et al., 2019).”
Conservative Q-Learning for Offline Reinforcement Learning (2020, methodology): “When only using only 1% or 10% of the data, in setting (2) (Table 3), CQL substantially outperforms REM and QR-DQN, especially in the harder 1% condition, achieving 36x and 6x times the return of the best prior method on Q*bert and Breakout, respectively.”
What Matters in Learning from Offline Human Demonstrations for Robot Manipulation (2021, methodology): “In Fig 4a, we show that this can be non-trivial, by showing the relative performance drop when selecting the policy using the best loss on validation data (common in supervised learning), and when using the final training checkpoint as well (common in offline RL [47, 18, 27]) – in both cases, the se”
Regularized Behavior Value Estimation (2021, methodology): “…et al., 2019b; Kumar et al., 2019; Jaques et al., 2019; Siegel et al., 2020; Wang et al., 2020; Ghasemipour et al., 2020), 2) value-based approaches lower value estimates for unseen state-actions pairs, either through regularization or uncertainty (Kumar et al., 2020; Agarwal et al., 2019a).”
Continuous Doubly Constrained Batch Reinforcement Learning (2021, methodology): “[26, 64] consider model-based approaches for batch RL, and [1] find ensembles partly address some of the issues that arise in batch settings.”
Provable Representation Learning for Imitation with Contrastive Fourier Features (2021, methodology): “We now study the practical beneﬁt of the proposed contrastive learning objective to imitation learning on 60 Atari 2600 games [11], taking for the ofﬂine dataset the DQN Replay Dataset [7], which for each game provides 50M steps collected during DQN training.”
COMBO: Conservative Offline Model-Based Policy Optimization (2021, methodology): “In the paper, we present conservative ofﬂine model-based policy optimization (COMBO), a model-based ofﬂine RL algorithm that penalizes the Q-values evaluated on out-of-support state-action pairs.”
Representation Matters: Offline Pretraining for Sequential Decision Making (2021, background): “Accordingly, much of the recent oﬄine RL research focuses on proposing new policy optimization algorithms amenable to learning from oﬄine datasets (e.g., Fujimoto et al. (2019); Wu et al. (2019); Agarwal et al. (2020); Kumar et al. (2020); Yu et al. (2020); Matsushima et al. (2020)).”
Hyperparameter Selection for Offline Reinforcement Learning (2020, methodology): “Recently, Agarwal et al. (2019), Fujimoto et al. (2019), Kumar et al. (2019) and Siegel et al. (2020) have proposed new oﬄine-RL algorithms and shown that they outperform oﬀ-the-shelf oﬀ-policy RL methods.”
RL Unplugged: Benchmarks for Offline Reinforcement Learning (2020, methodology): “It is a popular benchmark to measure the progress of online RL methods, and Atari has recently also become a standard benchmark for offline RL methods (Agarwal et al., 2020; Fujimoto et al., 2019) as well.”
Acme: A Research Framework for Distributed Reinforcement Learning (2020, methodology): “Our results closely matches the results reported in (Agarwal et al., 2019) for Offline DQN on most games with the same network architecture.”
TiKick: Towards Playing Multi-agent Football Full Games from Single-agent Demonstrations (2021, neutral): “…(Fujimoto, Meger, and Precup, 2019; Wu, Tucker, and Nachum, 2019), value regularization methods (Kumar et al., 2020) and uncertainty-based methods (Agarwal, Schuurmans, and Norouzi, 2020), there is no work on learning multiple cooperative agents with the incomplete (or partially observable)…”
QPLEX: Duplex Dueling Multi-Agent Q-Learning (2020, background): “Agarwal et al. (2020) presents an optimistic perspective of offline Q-learning that DQN and its variants can achieve superior performance in Atari 2600 games (Bellemare et al., 2013) with sufficiently large and diverse datasets.”
Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization (2020, background): “More recently, a related but distinct problem known as offline RL has gained popularity [33, 61, 1].”
Deep Reinforcement Learning at the Edge of the Statistical Precipice (2021, methodology): “References [1] Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi.”
Randomized Ensembled Double Q-Learning: Learning Fast Without a Model (2021, methodology): “REM uses the random convex combination of Q values to compute the target: it is similar to ensemble average (AVG), but with more randomization (Agarwal et al., 2020).”
Autoregressive Dynamics Models for Offline Policy Evaluation and Optimization (2021, background): “Even though offline RL offers the promise of leveraging existing logged datasets, current offline RL algorithms (Fujimoto et al., 2019; Agarwal et al., 2020; Kumar et al., 2019) are typically evaluated using online interaction, which limits their applicability in the real world.”