A Kernel Loss for Solving the Bellman Equation

By Yihao Feng, Lihong Li, Qiang Liu. 2019

75 citations recorded by the source corpus.

Value function learning plays a central role in many state-of-the-art reinforcement learning algorithms. Many popular algorithms like Q-learning do not optimize any objective function, but are fixed-point iterations of some variants of Bellman operator that are not necessarily a contraction. As a result, they may easily lose convergence guarantees, as can be observed in practice. In this paper, we propose a novel loss function, which can be optimized using standard gradient-based methods with guaranteed convergence. The key advantage is that its gradient can be easily approximated using sampled transitions, avoiding the need for double samples required by prior algorithms like residual gradient. Our approach may be combined with general function classes such as neural networks, using either on- or off-policy data, and is shown to work reliably and effectively in several benchmarks, including classic problems where standard algorithms are known to diverge.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Statistical and Nonlinear Physics, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sergey Levine
2. Ofir Nachum
3. Pieter Abbeel
4. Shixiang Shane Gu
5. Nan Jiang
6. Ilya Kostrikov
7. Bo Dai
8. George Tucker

In Computational Theory and Mathematics

1. Ofir Nachum
2. Nan Jiang
3. Jiawei Huang
4. Sergey Levine
5. Ziyang Tang
6. Shixiang Shane Gu
7. Lihong Li
8. Bo Dai

In Statistical and Nonlinear Physics

1. Ziyang Tang
2. Nan Jiang
3. Jiawei Huang
4. Lihong Li
5. Ofir Nachum
6. Doina Precup
7. Cong Ma
8. Bo Dai

In Reinforcement Learning

1. Sergey Levine
2. Ofir Nachum
3. Nan Jiang
4. Shixiang Shane Gu
5. Lihong Li
6. George Tucker
7. Ilya Kostrikov
8. Aviral Kumar

Citations in the corpus

• AlgaeDICE: Policy Gradient from Arbitrary Experience (2019, methodology): “There are several alternatives to handle this issue (e.g., Antos et al., 2008; Farahmand et al., 2016; Feng et al., 2019).”
• Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimation (2019, methodology): “Similarly, for the training of value function, we use primal-dual based optimization methods (Dai et al., 2017b; Feng et al., 2019) to minimize the bellman residual:”
• Harnessing Infinite-Horizon Off-Policy Evaluation: Double Robustness via Duality (2019, methodology): “Similarly, for the training of value function, we use primal-dual based optimization methods (Dai et al., 2017b; Feng et al., 2019) to minimize the bellman residual:”
• Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism (2021, methodology): “related algorithms in [66] and [67] use a milder definition based on a weighted norm of density ratios as opposed to the infinity norm.”
• Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Error (2022, methodology): “While much of the modern literature surrounding Bellman error minimization emphasizes the double sampling problem (Dai et al., 2018; Feng et al., 2019; Saleh & Jiang, 2019; Bas-Serrano et al., 2021), our analysis shows much more fundamental problems; Bellman error is not guaranteed to closely…”
• Minimax Confidence Interval for Off-Policy Evaluation and Policy Optimization (2020, methodology): “[14] Yihao Feng, Lihong Li, and Qiang Liu.”
• Q* Approximation Schemes for Batch Reinforcement Learning: A Theoretical Comparison (2020, background): “Another closely related work is kernel loss [Feng et al., 2019], which becomes similar to MABO when the implicit maximization in the RHKS is interpreted as searching over an importance weight class (this connection is pointed out by Uehara et al. [2019]).”