Non-delusional Q-learning and value-iteration

By Tyler Lu, Dale Schuurmans, Craig Boutilier. 2018

51 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Off-Policy Deep Reinforcement Learning without Exploration (2018, positive): “When combined with off-policy deep Q-learning to learn Q θ , this algorithm is referred to as Deep Deterministic Policy Gradients (DDPG) (Lillicrap et al., 2015).”
D4RL: Datasets for Deep Data-Driven Reinforcement Learning (2020, background): “Hand-designed controllers may not be representable by the learner’s policy class, introducing bias into the learning process [Lu et al., 2018].”
AlgaeDICE: Policy Gradient from Arbitrary Experience (2019, background): “However, such a justification is unsound in function approximation settings, where models have limited expressiveness, with potentially disastrous consequences on optimization and convergence (e.g., Lu et al., 2018).”
Information-Theoretic Considerations in Batch Reinforcement Learning (2019, background): “Related work that has not been covered The conjectured insufficiency of realizability (Conjecture 8) is related to various undesirable phenomena in learning with bootstrapped targets, which has been of constant interest to RL researchers (Sutton, 2015; Van Hasselt et al., 2018; Lu et al., 2018).”
A survey and critique of multiagent deep reinforcement learning (2018, background): “policy class”[80].”
Deep Reinforcement Learning (2018, background): “radient TD learning. White and White(2016) perform empirical comparisons of linear TD methods, and make suggestions about their practical use.Jin et al.(2018) study the sample efﬁciency of Q-learning.Lu et al. (2018) study non-delusional Q-learning and value-iteration. 19 algorithm TD() SARSA() ADP ”
Datasets for Data-Driven Reinforcement Learning (2020, background): “Hand-designed controllers may not be representable by the learner’s policy class, introducing bias into the learning process [Lu et al., 2018].”
Provably Good Batch Reinforcement Learning Without Great Exploration (2020, background): “One particular issue is that the max in the Bellman operator may pick actions in (s, a) pairs with limited but rewarding samples, which can lead to overly optimistic value function estimates and under-performing policies [26].”
ConQUR: Mitigating Delusional Bias in Deep Q-learning (2020, methodology): “7 illustrates the phenomenon: Lu et al. (2018) use a linear approximator over a speciﬁc set of features in this MDP to show that: (a) No π ∈ G (Θ) can express the optimal (unconstrained) policy (which requires taking a 2 at each state); (b) The optimal feasible policy in G (Θ) takes a 1 at s 1 and a”