Robust estimation of optimal dynamic treatment regimes for sequential treatment decisions

By Baqun Zhang, Anastasios A. Tsiatis, Eric B. Laber, Marie Davidian. 2013

226 citations recorded by the source corpus.

Summary A dynamic treatment regime is a list of sequential decision rules for assigning treatment based on a patient’s history. Q- and A-learning are two main approaches for estimating the optimal regime, i.e., that yielding the most beneficial outcome in the patient population, using data from a clinical trial or observational study. Q-learning requires postulated regression models for the outcome, while A-learning involves models for that part of the outcome regression representing treatment contrasts and for treatment assignment. We propose an alternative to Q- and A-learning that maximizes a doubly robust augmented inverse probability weighted estimator for population mean outcome over a restricted class of regimes. Simulations demonstrate the method’s performance and robustness to model misspecification, which is a key concern.

Subfields: Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistics and Probability

Citations in the corpus

Learning When-to-Treat Policies (2019, methodology): “Another approach to estimating Vπ is using a DR estimator as follows (Zhang et al. 2013; Jiang and Li 2016; Thomas and Brunskill 2016) V̂DRπ = 1 n n∑ i=1 ( γ̂ (i) T (π)Y (i) − T∑ t=1 ( γ̂ (i) t (π) − γ̂ (i)t−1(π) ) μ̂π ( S(i)1:t , A (i) 1:t−1 )) , (7) where μ̂π (·) is an estimator of μπ(·), the…”
Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processes (2019, methodology): “…model M1 is commonly the one assumed in the causal inference literature in the context of marginal structural model estimation (Robins, 2000; Robins et al., 2000) and DTRs (Chakraborty and Moodie, 2013; Murphy et al., 2001; Zhang et al., 2013),3 in RL one often assumes that the MDP model M2 holds.”
Estimating Dynamic Treatment Regimes in Mobile Health Using V-learning (2016, methodology): “Our approach, which we call V-learning, involves estimating the optimal policy among a prespeciﬁed class of policies (Zhang et al., 2012, 2013).”
New Statistical Learning Methods for Estimating Optimal Dynamic Treatment Regimes (2015, methodology): “In A-learning, proposed by Murphy (2003), one models regret functions which measure the loss incurred by not following the optimal treatment regime at each stage.”
Interactive Q-learning for Probabilities and Quantiles (2014, methodology): “We estimate the value using the Augmented Inverse Probability Weighted Estimator given in Zhang et al. (2013).”
Estimation of Optimal Dynamic Treatment Regimes (2014, methodology): “Methods for estimating V(d), include inverse probability-weighting [9,14]; augmented inverse probability-weighting [10,13]; and marginal structural mean models [7].”
Precision Medicine. (2019, background): “For these reasons, direct search estimators based on Vn(d) work best for settings where T is small, e.g., T = 2; however, in such settings OWL and related methods can offer significant gains in terms of robustness and marginal mean outcome (see Zhang et al 2013; Zhao et al 2015a).”
Incorporating Patient Preferences into Estimation of Optimal Individualized Treatment Rules (2017, background): “…2011; Goldberg and Kosorok, 2012; Moodie et al., 2012, 2014; Laber et al., 2014; Tian et al., 2014) or policy-search-based (Orellana et al., 2010; Zhang et al., 2012, 2013; Zhao et al., 2012, 2014, 2015; Laber and Zhao, 2015), though this division 18 © 2017, The International Biometric…”
Augmented outcome-weighted learning for estimating optimal dynamic treatment regimens (2018, background): “From the theory of Robins 8 , also seen in Tsiatis 21 and Zhang et al 22 , Q ik , the optimal reward increment for patient i if she/he follows the estimated optimal rule from stage k to K , has the following expression: , where m kj ( H ij ) is the optimal reward increment for subjects who receive…”
Robust Hybrid Learning for Estimating Personalized Dynamic Treatment Regimens (2016, methodology): “Zhang et al. (2012) and Zhang et al. (2013) developed a method based on augmented inverse probability weighting for a single or multiple stage model.”
Dynamic treatment regimes: technical challenges and applications (2014, methodology): “Recent statistical work in this area includes marginal structural mean models (Robins et al., 2008; Orellana et al., 2010), augmented value maximization (Zhang et al., 2012, 2013), and outcome weighted learning (Zhao et al., 2012, 2013).”
Estimating Optimal Treatment Regimes from a Classification Perspective (2012, methodology): “Under scenario 2 (shown in Table II) the true gopt is linear and not well-approximated by a tree with splits along the coordinate axes; the method of Zhang et al. (2012a) has the best performance, which is expected since the specified class G contains the true optimal regime.”
Q- and A-learning Methods for Estimating Optimal Dynamic Treatment Regimes (2012, background): “…(2002), Robins (2004), Moodie, Richardson and Stephens (2007), Thall et al. (2007), van der Laan and Petersen (2007), Robins, Orellana and Rotnitzky (2008), Almirall, Ten Have and Murphy (2010), Orellana, Rotnitzky and Robins (2010), Zhang et al. (2012a, 2012b, 2013) and Zhao et al. (2012, 2013).”
Interactive Q-learning for Quantiles (2017, methodology): “Non-parametric methods such as OWL (Zhao et al. 2012), double-robust direct search (Zhang et al. 2012a, 2012b, 2013), and boosting (Kang, Janes, and Huang 2014) are less restrictive than model-based approaches for optimizing binary outcomes, but the following exposition facilitates later…”
Efficient Evaluation of Natural Stochastic Policies in Offline Reinforcement Learning (2020, methodology): “In these domains, we must use offline RL, where we evaluate and learn new sequential decision policies from existing observational data (Murphy, 2003; Zhang et al., 2013; Bibaut et al., 2019; Kallus and Uehara, 2019c).”
Constructing Dynamic Treatment Regimes in Infinite-Horizon Settings (2014, unknown): “Their new method creates a pseudo population in which everyone has an equal number of decision points and they show that the results obtained by the pseudo population can be translated to the original problem (Zhao et al., 2011; Zhang et al., 2012, 2013).”