Offline Reinforcement Learning with Implicit Q-Learning

By Ilya Kostrikov, Ashvin Nair, Sergey Levine. International Conference on Learning Representations, 2021

1,586 citations recorded by the source corpus.

Offline reinforcement learning requires reconciling two conflicting aims: learning a policy that improves over the behavior policy that collected the dataset, while at the same time minimizing the deviation from the behavior policy so as to avoid errors due to distributional shift. This trade-off is critical, because most current offline reinforcement learning methods need to query the value of unseen actions during training to improve the policy, and therefore need to either constrain these actions to be in-distribution, or else regularize their values. We propose an offline RL method that never needs to evaluate actions outside of the dataset, but still enables the learned policy to improve substantially over the best behavior in the data through generalization. The main insight in our work is that, instead of evaluating unseen actions from the latest policy, we can approximate the policy improvement step implicitly by treating the state value function as a random variable, with randomness determined by the action (while still integrating over the dynamics to avoid excessive optimism), and then taking a state conditional upper expectile of this random variable to estimate the value of the best actions in that state. This leverages the generalization capacity of the function approximator to estimate the value of the best available action at a given state without ever directly querying a Q-function with this unseen action. Our algorithm alternates between fitting this upper expectile value function and backing it up into a Q-function. Then, we extract the policy via advantage-weighted behavioral cloning. We dub our method implicit Q-learning (IQL). IQL demonstrates the state-of-the-art performance on D4RL, a standard benchmark for offline reinforcement learning. We also demonstrate that IQL achieves strong performance fine-tuning using online interaction after offline initialization.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Reinforcement Learning

View Paper: https://arxiv.org/abs/2110.06169

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Reinforcement Learning

Citations in the corpus

Offline Reinforcement Learning as One Big Sequence Modeling Problem (2021, methodology): “We explore this combination by using the Q -function from the implicit Q -learning algorithm (IQL; Kostrikov et al. 2021) on the AntMaze navigation tasks (Fu et al., 2020), for which there is only a sparse reward upon reaching the goal state.”
Continuous Control with Action Quantization from Demonstrations (2021, positive): “We use the results reported by Kostrikov et al. (2021).”
BridgeData V2: A Dataset for Robot Learning at Scale (2023, neutral): “…assumptions: for example, reinforcement learning methods that require data with broader coverage and may benefit from noisy suboptimal trajectories [46], methods that require language descriptions to run language-conditioned imitation learning [5], or methods that use high-capacity models and…”
A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems (2022, methodology): “[23] propose using an expectile regression loss, so that we can think of V π ψ (s) as the best value from the actions within the support of our data.”
Efficient Online Reinforcement Learning with Offline Data (2023, background): “Again, IQL + Fine-tuning represents the strongest prior work (Kostrikov et al., 2022).”
Contrastive Learning as Goal-Conditioned Reinforcement Learning (2022, methodology): “Lastly, we compare with two more complex methods that use TD learning: TD3+BC [39] and IQL [68].”
Planning with Diffusion for Flexible Behavior Synthesis (2022, methodology): “…compare to a variety of prior algorithms spanning other approaches to data-driven control, including the model-free reinforcement learning algorithms CQL (Kumar et al., 2020) and IQL (Kostrikov et al., 2022) (Yu et al., 2020), MOReL (Kidambi et al., 2020), and MBOP (Argenson & Dulac-Arnold, 2021).”
Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons (2023, methodology): “As an alternative, there has been a number of heuristic approximations considered, including Conservative Q-Learning (Kumar et al., 2020), Implicit Q-Learning (Kostrikov et al., 2021) and Adversarially Trained Actor Critic (Cheng et al., 2022).”
Offline RL for Natural Language Generation with Implicit Language Q Learning (2022, methodology): “ILQL builds on the IQL algorithm, extending it to the token-level POMDP that defines NLP tasks via the following modifications: (i) it integrates with sequence models to handle partially observable language generation tasks (Section 4.2); (ii) it utilizes a novel policy extraction method that direct”
Is Conditional Generative Modeling all you need for Decision-Making? (2022, methodology): “We compare with existing offline RL methods, including model-free algorithms like CQL (Kumar et al., 2020) and IQL (Kostrikov et al., 2022), and model-based algorithms such as trajectory transformer ( TT , Janner et al. (2021)) and MoReL (Kidambi et al., 2020).”
Bisimulation Makes Analogies in Goal-Conditioned Reinforcement Learning (2022, methodology): “All other baselines compared to in Section 7 allow critic gradients from IQL’s Q function into the encoder.”