Interactive model building for Q-learning

By Eric B. Laber, Kristin A. Linn, Leonard A. Stefanski. 2014

78 citations recorded by the source corpus.

Summary Evidence-based rules for optimal treatment allocation are key components in the quest for efficient, effective health care delivery. Q-learning, an approximate dynamic programming algorithm, is a popular method for estimating optimal sequential decision rules from data. Q-learning requires the modeling of nonsmooth, nonmonotone transformations of the data, complicating the search for adequately expressive, yet parsimonious, statistical models. The default Q-learning working model is multiple linear regression, which is not only provably misspecified under most data-generating models, but also results in nonregular regression estimators, complicating inference. We propose an alternative strategy for estimating optimal sequential decision rules for which the requisite statistical modeling does not depend on nonsmooth, nonmonotone transformed data, does not result in nonregular regression estimators, is consistent under a broader array of data-generation models than Q-learning, results in estimated sequential decision rules that have better sampling properties, and is amenable to established statistical approaches for exploratory data analysis, model building, and validation. We derive the new method, IQ-learning, via an interchange in the order of certain steps in Q-learning. In simulated experiments IQ-learning improves on Q-learning in terms of integrated mean squared error and power. The method is illustrated using data from a study of major depressive disorder.

Subfields: Artificial Intelligence, Economics and Econometrics, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Economics and Econometrics

In Statistics and Probability

Citations in the corpus

Reinforcement Learning in Healthcare: A Survey (2019, methodology): “regularity of the parameter estimations in traditional RL methods by proposing various extensions over default Q-learning procedure in order to increase the robustness of learning [167]. Laber et al. [168] proposed a new version of Q-learning, interactive Qlearning (IQ-learning), by interchanging th”
Using Decision Lists to Construct Interpretable and Parsimonious Treatment Regimes (2015, background): “Thus, a goal of many regression-based estimators is to ensure correct model specification under a large class of generative models (Zhao et al., 2009; Qian and Murphy, 2011; Moodie et al., 2013; Laber et al., 2014; Taylor et al., 2014).”
Q-Learning: Theory and Applications (2020, methodology): “In addition to problems of statistical inference, developments for finite Q-learning in statistics have included extensions to nonlinear model classes (Zhao et al. 2009, Laber et al. 2014a, Moodie et al. 2014, Zhang et al. 2015, Xu et al. methods (Zhang et al. 2012, Zhang et al. 2015).”
Dynamic treatment regimes: technical challenges and applications (2014, methodology): “(The three-treatment version of the ACI is given by Laber et al. [2010].) Note that given the working models and generative models defined by the parameter settings in Table 10, we can determine the exact value of any parameter cTβ∗ 1 of interest to set the ground truth for our experiments.”
Interactive Q-learning for Quantiles (2017, methodology): “Interactive Q -learning (IQ-learning), developed for the two-stage binary treatment setting, requires modeling only smooth, monotone transformations of the data, thereby reducing problems of model misspec-ification and nonregular inference (Laber, Linn, and Stefanski 2014a).”
Q-learning Residual Analysis: Application to The Effectiveness of Sequences of Antipsychotic Medications for Patients with Schizophrenia (2016, background): “[29] proposed a version of Q-learning called interactive Q-learning that replaces modeling the predicted future optimal outcomes Ỹ (i.”
Incorporating Patient Preferences into Estimation of Optimal Individualized Treatment Rules (2017, background): “Set-valued treatment regimes (Laber et al., 2014; Lizotte and Laber, 2016) map current patient information to a subset of possible treatment options that are not estimated to be uniformly worse across all outcomes; the expectation is that a single treatment will be selected from the recommended set…”
Estimating Dynamic Treatment Regimes in Mobile Health Using V-learning (2016, methodology): “Furthermore, GGQ requires modeling a non-smooth function of the data, which creates complications (Laber et al., 2014; Linn et al., 2017).”
Interactive Q-learning for Probabilities and Quantiles (2014, methodology): “Interactive Q-learning (IQ-learning), developed for the two-stage binary treatment setting, requires modeling only smooth, monotone transformations of the data, thereby reducing problems of model misspecification and nonregular inference (Laber et al., 2014).”
Interpretable Dynamic Treatment Regimes (2016, methodology): “…that even under the simplest generative models the optimal regime is a nonlinear function of patient information (Robins 2004; Schulte et al. 2014; Laber, Linn, and Stefanski 2014); consequently, to avoid model misspecification, a recent trend is to apply flexible supervised learning methods to…”
Adaptive Interventions Treatment Modelling and Regimen Optimization Using Sequential Multiple Assignment Randomized Trials (Smart) and Q-Learning (2018, background): “in 2014, for adequate fitting and well interpretable model [14].”
Precision Medicine. (2019, methodology): “…Shortreed and Moodie 2012; Shortreed et al 2014; Kosorok and Moodie 2016); measurement error (Shani et al 2013), and model-building (Biernot and Moodie 2010; Rich et al 2010; Henderson et al 2010; Gunter et al 2011; Lu et al 2013; Laber et al 2014; Song et al 2015; Luedtke and van der Laan 2016b).”