Linear Fitted-Q Iteration with Multiple Reward Functions

By Daniel J. Lizotte, Michael H. Bowling, Susan A. Murphy. 2012

99 citations recorded by the source corpus.

We present a general and detailed development of an algorithm for finite-horizon fitted-Q iteration with an arbitrary number of reward signals and linear value function approximation using an arbitrary number of state features. This includes a detailed treatment of the 3-reward function case using triangulation primitives from computational geometry and a method for identifying globally dominated actions. We also present an example of how our methods can be used to construct a real-world decision aid by considering symptom reduction, weight gain, and quality of life in sequential treatments for schizophrenia. Finally, we discuss future directions in which to take this work that will further enable our methods to make a positive impact on the field of evidence-based clinical decision support.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Ordered Preference Elicitation Strategies for Supporting Multi-Objective Decision Making (2018, background): “The AI subfield of multi-objective decision theoretic planning and learning [3, 31, 34, 35, 47, 49–51] studies complex multi-objective decision problems.”
A Survey of Multi-Objective Sequential Decision-Making (2013, methodology): “To our knowledge, the only work in this area is that of Lizotte et al. (2010, 2012), where a model of the MOMDP’s transition probabilities and reward function is derived from historical data, and then a spline-based multi-objective value iteration approach is applied to that model.”
Incorporating Patient Preferences into Estimation of Optimal Individualized Treatment Rules (2017, background): “The assumption of a linear utility is common in multiobjective optimization (Fishburn, 1988; Marler and Arora, 2004; Richter and Wong, 2004; Lizotte et al., 2012).”
Multi-Objective Markov Decision Processes for Data-Driven Decision Support (2016, methodology): “, 2011) and examining convex combinations of basis rewards (Lizotte et al., 2012).”
Interactive model building for Q-learning (2014, background): “…a measure of efficacy, the Quick Inventory of Depression Symptomatology score, as the outcome (Rush et al., 2004); side-effects or other competing outcomes could be accommodated using set-valued treatment regimes (Lizotte et al., 2012; Laber et al., 2013) or composite outcomes (Wang et al., 2012).”
Set-valued dynamic treatment regimes for competing outcomes (2012, methodology): “We transformed both outcomes into percentiles to match Lizotte et al. (2012); we also include an analysis using the raw BMI and PANSS scores in Web Appendix C. Regression diagnostics for both analyses as well as baseline distributions are given in Web Appendix D.”
Optimization of anemia treatment in hemodialysis patients via reinforcement learning (2014, background): “The model introduced in this section is focused in patients undergoing hemodialysis who are treated with intravenous darbepoetinalfa, a second generation ESA drug.”
Precision Medicine. (2019, background): “…with multiple outcomes includes set-valuated treatment regimes which recommend a set of ‘acceptable’ treatments given a patient’s history (Fard 2009; Lizotte et al 2012; Laber et al 2014; Lizotte and Laber 2016; Wu 2016); and methods where a primary outcome is maximized while a secondary outcome…”
Estimation of Optimal Dynamic Treatment Regimes (2014, methodology): “Other advantages of Q-learning are as follows: (1) it can be extended to accommodate discrete outcomes [45,50], censored outcomes [49], and competing outcomes [54,55]; (2) measures of goodness of fit and visual diagnostics can be used to assess the quality of the fitted regression models in each sta”
Multi-objective reinforcement learning using sets of pareto dominating policies (2014, background): “Recently, they validated their work on clinical trial data for three objectives, although the practical possibilities for higher dimensional spaces are not straightforward (Lizotte et al., 2012).”