Q- and A-learning Methods for Estimating Optimal Dynamic Treatment Regimes

By Phillip J. Schulte, Anastasios A. Tsiatis, Eric B. Laber, Marie Davidian. 2012

240 citations recorded by the source corpus.

In clinical practice, physicians make a series of treatment decisions over the course of a patient’s disease based on his/her baseline and evolving characteristics. A dynamic treatment regime is a set of sequential decision rules that operationalizes this process. Each rule corresponds to a decision point and dictates the next treatment action based on the accrued information. Using existing data, a key goal is estimating the optimal regime, that, if followed by the patient population, would yield the most favorable outcome on average. Q- and A-learning are two main approaches for this purpose. We provide a detailed account of these methods, study their performance, and illustrate them using data from a depression study.

Subfields: Management Science and Operations Research, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Management Science and Operations Research

In Statistics and Probability

Citations in the corpus

Precision Medicine. (2019, methodology): “…interventions at any given time point may vary according to a patient’s health status, availability, or other factors (Bembom and van der Laan 2008; Schulte et al 2014; Laber and Staicu In press; Laber et al 2018); however, for simplicity, in describing methods for estimation and inference in…”
Interpretable Dynamic Treatment Regimes (2016, methodology): “linical outcome if applied to a population of interest. It can be shown that even under the simplest generative models the optimal regime is a nonlinear function of patient information (Robins, 2004; Schulte et al., 2014; Laber et al., 2014); consequently, to avoid model misspecication, a recent tre”
New Statistical Learning Methods for Estimating Optimal Dynamic Treatment Regimes (2015, background): “For more discussion on the relationship between Q- and A-learning see Schulte et al. (2014).”
Estimating Optimal Treatment Regimes from a Classification Perspective (2012, methodology): “which is exactly the approximated weighted classification error used by Zhao et al. (2012). Zhao et al. (2012) minimize the above weighted classification error using support vector machines.”
Interactive Q-learning for Quantiles (2017, methodology): “In addition, it has been shown that Q -learning, A -learning, and g -estimation lead to identical estimators in certain cases but that Q -learning is less efficient (Chakraborty, Murphy, and Strecher 2010; Schulte et al. 2012).”
Adaptive Interventions Treatment Modelling and Regimen Optimization Using Sequential Multiple Assignment Randomized Trials (Smart) and Q-Learning (2018, methodology): “in 2014 implemented Q and A learning methods to estimate the optimal treatment regimens and the contrast between these two statistical estimation methods was also discussed [16].”
Interactive model building for Q-learning (2014, methodology): “While we focus on randomized studies, the proposed approach is valid for observational data under the same causal conditions required for Q-learning (Schulte et al., 2012).”
Constructing Dynamic Treatment Regimes in Infinite-Horizon Settings (2014, unknown): “The treatment regime (policy), π, is a deterministic decision rule where for every s, the output π(s) is an action a ∈ As, where As is the space of feasible actions (Robins, 2004; Schulte et al., 2014).”
Incorporating Patient Preferences into Estimation of Optimal Individualized Treatment Rules (2017, unknown): “Under (C1)-(C3), it can be shown (Schulte et al., 2014) that π(x, w) = arg maxa∈{−1,1} E [ (H)Y + { 1 − (H)}Z∣∣X = x, W = w, A = a] which, under (C4), can be written as”
A Theoretical Analysis of Deep Q-Learning (2019, background): “Murphy (2003); Zhao et al. (2009); Qian and Murphy (2011); Zhao et al. (2011); Zhang et al. (2012); Zhao et al. (2012); Goldberg and Kosorok (2012); Nahum-Shani et al. (2012); Goldberg et al. (2013); Schulte et al. (2014); Song et al. (2015);Zhao et al. (2015); Linn et al. (2017);Zhou et al. (2017);”
Interactive Q-learning for Probabilities and Quantiles (2014, methodology): “Although our developments are made under the assumption of treatment randomization, additional assumptions about the treatment assignment mechanism allow application of the proposed methods to observational data (Murphy, 2003; Robins, 2004; Moodie et al., 2012; Schulte et al., 2014).”
Estimating Dynamic Treatment Regimes in Mobile Health Using V-learning (2016, methodology): “The goal is to select treatments to maximize expected cumulative utility; treatment selection is formalized using a treatment regime (Schulte et al. 2014; Kosorok and Moodie 2015) and the utility associated with any regime is defined using potential outcomes (Rubin 1978).”