Estimating Dynamic Treatment Regimes in Mobile Health Using V-learning

By Daniel J. Luckett, Eric B. Laber, A. Kahkoska, David M. Maahs, E. Mayer‐Davis, Michael R. Kosorok. 2016

164 citations recorded by the source corpus.

Abstract The vision for precision medicine is to use individual patient characteristics to inform a personalized treatment plan that leads to the best possible healthcare for each patient. Mobile technologies have an important role to play in this vision as they offer a means to monitor a patient’s health status in real-time and subsequently to deliver interventions if, when, and in the dose that they are needed. Dynamic treatment regimes formalize individualized treatment plans as sequences of decision rules, one per stage of clinical intervention, that map current patient information to a recommended treatment. However, most existing methods for estimating optimal dynamic treatment regimes are designed for a small number of fixed decision points occurring on a coarse time-scale. We propose a new reinforcement learning method for estimating an optimal treatment regime that is applicable to data collected using mobile technologies in an outpatient setting. The proposed method accommodates an indefinite time horizon and minute-by-minute decision making that are common in mobile health applications. We show that the proposed estimators are consistent and asymptotically normal under mild conditions. The proposed methods are applied to estimate an optimal dynamic treatment regime for controlling blood glucose levels in patients with type 1 diabetes.

Subfields: Endocrinology, Diabetes and Metabolism, General Health Professions, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Endocrinology, Diabetes and Metabolism

In General Health Professions

In Statistics and Probability

Citations in the corpus

Reinforcement Learning in Healthcare: A Survey (2019, methodology): “[132] proposed the V-learning method that directly estimates a”
Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learning (2019, background): “Luckett et al. (2018); Ueno et al. (2011) considered related semiparametric estimation techniques for the v -function.”
Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processes (2019, positive): “Although we do not explicitly use counterfactual notation (either potential outcomes or do-calculus), if we assume the usual sequential ignorability conditions (Ertefaie and Strawderman, 2018; Luckett et al., 2018; Murphy et al., 2001), the estimands we consider are the same and our results immediat”
Efficiently Breaking the Curse of Horizon: Double Reinforcement Learning in Infinite-Horizon Processes (2019, background): “Luckett et al. (2018); Ueno et al. (2011) considered related semiparametric estimation techniques for the v -function.”
Efficient Evaluation of Natural Stochastic Policies in Offline Reinforcement Learning (2020, methodology): “Note that the regressions can also be replaced with Z-estimation (Ueno et al., 2011; Luckett et al., 2018) based on the moment equations: E[gt(st, at) (rt + qt+1(st+1, π)− qt(st, at))] = 0, ∀gt, t ≤ H.”
Precision Medicine. (2019, methodology): “However, cross-pollination is rapidly increasing due to technological advancements facilitating the collection and curation of massive amounts of patient-level data in real (or near-real) time and the emergence of mobile-health (Tewari and Murphy 2017; Nahum-Shani et al 2017; Luckett et al 2017).”