A Generalization Error for Q-Learning

By Susan A. Murphy. 2005

241 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Learning When-to-Treat Policies (2019, methodology): “We formulate causal effects in terms of potential outcomes (Neyman 1923; Rubin 1974; Robins 1986).”
Off-Policy Evaluation via Off-Policy Classification (2019, methodology): “� to the sum of advantages t t A ⇡ ( s , a ) over data from ⇡ b , and was proposed by Kakade & Langford [14] and Murphy [26].”
Reinforcement Learning Strategies for Clinical Trials in Non-small Cell Lung Cancer (2011, background): “This design enables estimation of optimal individualized treatment regimes defined in counterfactual settings (cf. Murphy, 2005a).”
PERFORMANCE GUARANTEES FOR INDIVIDUALIZED TREATMENT RULES (2011, methodology): “Murphy [21] showed that Intuitively, this upper bound means that if the excess prediction error of Q [i.e., L ( Q ) − L ( Q 0 )] is small, then the reduction in Value of the associated ITR d [i.e., V ( d 0 ) − V ( d )] is small.”
Quasi-oracle estimation of heterogeneous treatment effects (2017, background): “This problem is closely related to, but subtly diﬀerent from the problem of estimating τ ∗ ( · ) under squared-error loss; see Kitagawa and Tetenov (2018), Manski (2004) and Murphy (2005) for a discussion.”
Reinforcement Learning and Dynamic Programming Using Function Approximators (2010, methodology): “Online algorithms, mainly approximate versions of Q-learning, have been studi ed s nce the beginning of the nineties (Lin, 1992; Singh et al., 1995; Horiuchi et al ., 1996; Jouffe, 1998; Glorennec, 2000; Tuyls et al., 2002; Szepesvári and Smart, 2004; Murphy, 2005; Sherstov and Stone, 2005; Melo et ”
New Statistical Learning Methods for Estimating Optimal Dynamic Treatment Regimes (2015, methodology): “Dynamic treatment regimes (DTRs), also called adaptive treatment strategies (Murphy 2003, 2005a), are sequential de- Ying-Qi Zhao is Assistant Professor, Department of Biostatistics and Medical Informatics, University of Wisconsin-Madison, WI 53792 (E-mail: yqzhao@biostat.wisc.edu), Donglin Zeng is…”
Interactive Q-learning for Probabilities and Quantiles (2014, methodology): “Current methods for estimating optimal dynamic treatment regimes include Q-learning (Watkins, 1989; Watkins and Dayan, 1992; Murphy, 2005a), regularized Q-learning (Moodie and Richardson, 2010; Chakraborty et al.”
Near-Optimal Reinforcement Learning in Dynamic Treatment Regimes (2019, methodology): “The optimal policy is thus solvable using the standard off-policy learning methods such as Q-learning [31, 20].”
Interactive Q-learning for Quantiles (2017, methodology): “Examples of regression-based estimators include Q -learning (Watkins 1989; Watkins and Dayan 1992; Murphy 2005a), regularized Q -learning (Chakraborty, Murphy, and Strecher 2010; Moodie and Richardson 2010; Goldberg, Song, and Kosorok 2013; Song et al. 2015), Interactive Q -learning (Laber, Linn,…”
Estimating the optimal dynamic antipsychotic treatment regime: Evidence from the sequential multiple assignment randomized CATIE Schizophrenia Study (2012, methodology): “However, proposed improvements to the basic Q-learning algorithm (Chakraborty et al., 2010) require additional programming and the basic framework is likely to be less familiar to statisticians who are not actively working in the area of dynamic treatment regimes.”
Learning near-optimal policies with Bellman-residual minimization based fitted policy iteration and a single sample path (2006, background): “In fact, the only paper where finite-sample bounds are derived in an off-policy setting and which uses function approximators is the paper by Murphy (2005) who considered fitted Q-iteration for finite-horizon, undiscounted problems.”
Dynamic treatment regimes: technical challenges and applications (2014, methodology): “In the two stage setting, the Q-functions (Sutton and Barto, 1998; Murphy, 2005b) are Q2(h2, a2) , E(Y |H2 = h2, A2 = a2), Q1(h1, a1) , E ( max a2 Q2(H2, a2) ∣∣H1 = h1, A1 = a1 ) , (1) so that Q2(h2, a2) measures the quality of assigning treatment a2 to a patient presenting with h2 at the second…”
Finite time bounds for sampling based fitted value iteration (2005, background): “The results in (Murphy, 2005) are actually presented for partially observable problems.”
Q-Learning: Theory and Applications (2020, methodology): “That is, letting (cid:16) π n be an estimator of π opt based on a SMART of sample size n , and letting B 0 > 0, γ , α , η , (cid:19) , ζ ∈ (0,1) be constants, they derive procedures for choosing n such that (POW) There exists an α -level test of H : V ( π opt ) ≤ B based on (cid:16) π that has power”
Incorporating Patient Preferences into Estimation of Optimal Individualized Treatment Rules (2017, background): “Estimators are typically broadly categorized as regression-based (Murphy, 2005; Henderson et al., 2010; Qian and Murphy, 2011; Zhao et al., 2011; Goldberg and Kosorok, 2012; Moodie et al., 2012, 2014; Laber et al., 2014; Tian et al., 2014) or policy-search-based (Orellana et al.”