Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning
By Yue Wu, Shuangfei Zhai, Nitish Srivastava, J. Susskind, Jian Zhang, Ruslan Salakhutdinov, Hanlin Goh. 2021
230 citations recorded by the source corpus.
Offline Reinforcement Learning promises to learn effective policies from previously-collected, static datasets without the need for exploration. However, existing Q-learning and actor-critic based off-policy RL algorithms fail when bootstrapping from out-of-distribution (OOD) actions or states. We hypothesize that a key missing ingredient from the existing methods is a proper treatment of uncertainty in the offline setting. We propose Uncertainty Weighted Actor-Critic (UWAC), an algorithm that detects OOD state-action pairs and down-weights their contribution in the training objectives accordingly. Implementation-wise, we adopt a practical and effective dropout-based uncertainty estimation method that introduces very little overhead over existing RL algorithms. Empirically, we observe that UWAC substantially improves model stability during training. In addition, UWAC out-performs existing offline RL methods on a variety of competitive tasks, and achieves significant performance gains over the state-of-the-art baseline on datasets with sparse demonstrations collected from human experts.
Subfields: Artificial Intelligence, Cognitive Neuroscience
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Chelsea Finn
- Pieter Abbeel
- Tianhe Yu
- Aravind Rajeswaran
- Aviral Kumar
- Benjamin Eysenbach
- Rafael Rafailov
In Cognitive Neuroscience
- Chelsea Finn
- Aviral Kumar
- Tianhe Yu
- Sergey Levine
- Benjamin Eysenbach
- Aravind Rajeswaran
- Ilya Kostrikov
- Rafael Rafailov
Citations in the corpus
- COMBO: Conservative Offline Model-Based Policy Optimization (2021, methodology): “…implicitly via regularized variants of importance sampling based algorithms [47, 58, 35, 59, 41], offline actor-critic methods [53, 45, 27, 16, 64], applying uncertainty quantification to the predictions of the Q-values [2, 28, 63, 34], and learning conservative Q-values [29, 55] or explicitly…”
- RvS: What is Essential for Offline RL via Supervised Learning? (2021, methodology): “Recent offline RL methods use many algorithmic techniques, including value functions [15, 27], dynamics estimation alongisde value functions [24, 37, 42], dynamics estimation alone [3, 7, 39], and uncertainty quantification [1, 41, 42].”
- Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (2023, methodology): “In practice, we typically resort to heuristic methods (Wu et al., 2021b; Coste et al., 2023) to estimate the uncertainty.”