Is the Bellman residual a bad proxy?

25 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Diagnosing Bottlenecks in Deep Q-learning Algorithms (2019, neutral): “(Geist et al., 2017) suggests that when the state-distribution is ﬁxed, the action distribution should be weighted by the optimal policy for residual Bellman errors.”
Non-delusional Q-learning and value-iteration (2018, background): “|| || � As such, minimizing BE does not generally provide policy guarantees of practical import (see, e.g., [11]).”
Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism (2021, background): “For a stationary deterministic optimal policy, C := C ? is the “best” concentrability coefficient definition [42, 13, 2, 52] which is often much smaller than the widely-used uniform concentrability coefficient C := maxπ C which takes the maximum over all policies.”
Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Error (2022, background): “Geist et al. (2017) evaluate the Bellman error as an objective for policy optimization.”
Q* Approximation Schemes for Batch Reinforcement Learning: A Theoretical Comparison (2020, methodology): “As another example, linear-in-horizon error can be achieved if one can directly minimize the Bellman error [e.g., Geist et al., 2017], but computing that requires knowledge of the transition probabilities.”