Stochastic Primal-Dual Q-Learning

By Donghwan Lee, Niao He. 2018

17 citations recorded by the source corpus.

In this work, we present a new model-free and off-policy reinforcement learning (RL) algorithm, that is capable of finding a near-optimal policy with state-action observations from arbitrary behavior policies. Our algorithm, called the stochastic primal-dual Q-learning (SPD Q-learning), hinges upon a new linear programming formulation and a dual perspective of the standard Q-learning. In contrast to previous primal-dual RL algorithms, the SPD Q-learning includes a Q-function estimation step, thus allowing to recover an approximate policy from the primal solution as well as the dual solution. We prove a first-of-its-kind result that the SPD Q-learning guarantees a certain convergence rate, even when the state-action distribution is time-varying but sub-linearly converges to a stationary distribution. Numerical experiments are provided to demonstrate the off-policy learning abilities of the proposed algorithm in comparison to the standard Q-learning.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Electrical and Electronic Engineering, Reinforcement Learning, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sergey Levine
2. Ofir Nachum
3. Aviral Kumar
4. George Tucker
5. Chelsea Finn
6. Pieter Abbeel
7. Ilya Kostrikov
8. Shixiang Shane Gu

In Computational Theory and Mathematics

1. Sergey Levine
2. Ofir Nachum
3. Aviral Kumar
4. Chelsea Finn
5. George Tucker
6. Justin Fu
7. Bo Dai
8. Ilya Kostrikov

In Electrical and Electronic Engineering

1. Sergey Levine
2. Ofir Nachum
3. Aviral Kumar
4. George Tucker
5. Chelsea Finn
6. Justin Fu
7. Ilya Kostrikov
8. Dale Schuurmans

In Reinforcement Learning

1. Sergey Levine
2. Ofir Nachum
3. Aviral Kumar
4. George Tucker
5. Chelsea Finn
6. Ilya Kostrikov
7. Justin Fu
8. Pieter Abbeel

In Optimization & Training

1. Chelsea Finn
2. Sergey Levine
3. Aviral Kumar
4. George Tucker
5. Justin Fu
6. Dale Schuurmans
7. Ofir Nachum
8. Bo Dai

Citations in the corpus

• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, background): “Lee and He (2018) extend a line of work applying to convex optimization techniques to policy optimization (Chen and Wang, 2016; Wang and Chen, 2016; Dai et al., 2017a,b) to the off-policy setting.”
• AlgaeDICE: Policy Gradient from Arbitrary Experience (2019, background): “Several recent works (e.g., Chen and Wang, 2016; Wang, 2017; Dai et al., 2018a,b; Chen et al., 2018; Lee and He, 2018) also considered saddle-point formulations for policy improvement, derived from fundamentally different perspectives.”
• Applications of reinforcement learning in energy systems (2021, background): “[187] extends it to the Q-learning framework with off-policy learning.”
• Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models (2023, methodology): “Our approach is also related to several video prediction methods [17, 38, 3, 58] but utilizes a better neural network architecture (i.”
• Stochastic Primal-Dual Q-Learning Algorithm For Discounted MDPs (2019, background): “Finally, we defer all proofs in this paper to its full online version [27] due to the space limitation.”