IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies
By Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, Sergey Levine. arXiv.org, 2023
320 citations recorded by the source corpus.
Effective offline RL methods require properly handling out-of-distribution actions. Implicit Q-learning (IQL) addresses this by training a Q-function using only dataset actions through a modified Bellman backup. However, it is unclear which policy actually attains the values represented by this implicitly trained Q-function. In this paper, we reinterpret IQL as an actor-critic method by generalizing the critic objective and connecting it to a behavior-regularized implicit actor. This generalization shows how the induced actor balances reward maximization and divergence from the behavior policy, with the specific loss choice determining the nature of this tradeoff. Notably, this actor can exhibit complex and multimodal characteristics, suggesting issues with the conditional Gaussian actor fit with advantage weighted regression (AWR) used in prior methods. Instead, we propose using samples from a diffusion parameterized behavior policy and weights computed from the critic to then importance sampled our intended policy. We introduce Implicit Diffusion Q-learning (IDQL), combining our general IQL critic with the policy extraction method. IDQL maintains the ease of implementation of IQL while outperforming prior offline RL methods and demonstrating robustness to hyperparameters. Code is available at https://github.com/philippe-eecs/IDQL.
Subfields: Experimental and Cognitive Psychology, Statistics and Probability
View Paper: https://www.semanticscholar.org/paper/a6f9fb141034a87ff9d627dc8a3ef31d0790c6ed
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Experimental and Cognitive Psychology
In Statistics and Probability
Citations in the corpus
- BridgeData V2: A Dataset for Robot Learning at Scale (2023, methodology): “Prior work has shown that highly expressive policies can better model multi-modal action distributions, leading to improved performance on a variety of tasks [50, 18, 47].”
- NoMaD: Goal Masked Diffusion Policies for Navigation and Exploration (2023, unknown): “…this by exploring different action representations, such as autoregressive predictions of quantized actions [24–27], using latent variable models [17, 21], switching to an implicit policy representation [28], and, most recently, using conditional diffusion models for planning and control [29–34].”
- 3D Diffuser Actor: Policy Diffusion with 3D Scene Representations (2024, unknown): “They have also been used successfully in offline reinforcement learning [7, 82, 24], where they model the state-conditioned action trajectory distribution [24, 7] or state-action trajectory distribution [35, 26, 79].”
- Training Diffusion Models with Reinforcement Learning (2023, unknown): “Their flexibility in representing complex, high-dimensional distributions has led to the adoption of diffusion models in applications including image and video synthesis [38, 41, 20], drug and material design [56, 55, 42], and continuous control [21, 52, 15].”
- Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models (2023, methodology): “We use a diffusion model for our goal-reaching policy since recent work has shown that diffusion-based policies can better capture multi-modality in robot data [12, 23], leading to improved performance across a variety of tasks.”