Balanced Policy Evaluation and Learning

By Nathan Kallus. 2017

162 citations recorded by the source corpus.

We present a new approach to the problems of evaluating and learning personalized decision policies from observational data of past contexts, decisions, and outcomes. Only the outcome of the enacted decision is available and the historical policy is unknown. These problems arise in personalized medicine using electronic health records and in internet advertising. Existing approaches use inverse propensity weighting (or, doubly robust versions) to make historical outcome (or, residual) data look like it were generated by a new policy being evaluated or learned. But this relies on a plug-in approach that rejects data points with a decision that disagrees with the new policy, leading to high variance estimates and ineffective learning. We propose a new, balance-based approach that too makes the data look like the new policy but does so directly by finding weights that optimize for balance between the weighted data and the target policy in the given, finite sample, which is equivalent to minimizing worst-case or posterior conditional mean square error. Our policy learner proceeds as a two-level optimization problem over policies and weights. We demonstrate that this approach markedly outperforms existing ones both in evaluation and learning, which is unsurprising given the wider support of balance-based weights. We establish extensive theoretical consistency guarantees and regret bounds that support this empirical success.

Subfields: Management Science and Operations Research, Statistics and Probability, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Management Science and Operations Research

In Statistics and Probability

In Reinforcement Learning

Citations in the corpus

Balanced Linear Contextual Bandits (2018, methodology): “Finally, note that one could integrate in the contextual bandit estimation other covariate balancing methods, such as the method of approximate residual balancing (Athey, Imbens, and Wa-ger 2018) or the method of (Kallus 2017).”
Estimation Considerations in Contextual Bandits (2017, methodology): “Finally, note that one could integrate in the contextual bandit estimation other covariate balancing methods, such as the method of approximate residual balancing [8] or the method of [38].”
Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learning (2019, methodology): “Equations (2) and (3) are almost the same as the limit as T →∞ of cT (γ) times the finite-horizon efficiency bounds derived by Kallus and Uehara (2019a). They are the same if we replace the lower summation limit with k = 0 instead of k = 1 in Eqs.”
Intrinsically Efficient, Stable, and Bounded Off-Policy Evaluation for Reinforcement Learning (2019, background): “The weights ω0:t are assumed known here as is common in RL; otherwise they can either be estimated directly or chosen by optimal balance [1, 8, 9].”
Learning When-to-Treat Policies (2019, background): “There is also a growing number of existing works that have applied orthogonal moments construction to policy evaluation (Belloni et al., 2017; Chernozhukov et al., 2016b; Kallus, 2018).”
Efficient Policy Learning (2017, background): “The only other available 1/ √ n-bounds for policy learning in observational studies with a binary treatment that we are aware of are a result of van der Laan, Dudoit, and van der Vaart (2006) for the case where Π consists of a finite set of policies whose cardinality grows with n, and a result of Ka”
Statistically Efficient Off-Policy Policy Gradients (2020, unknown): “This asymptotic results can also be extended to a finite-sample results by assuming finite-sample bounds on nuisance estimates, following Kallus & Uehara (2019b).”
Fairness in Machine Learning: A Survey (2020, background): “[163, 276, 177, 178, 153] note that there is an additional challenge in the datasets used to train models: data represent past decisions, and as such inherent bias(es) in these decisions are amplified.”
Policy Learning With Observational Data (2017, background): “…in both statistics (Luedtke and van der Laan, 2016; Qian and Murphy, 2011; Zhang, Tsiatis, Davidian, Zhang, and Laber, 2012; Zhao, Zeng, Rush, and Kosorok, 2012) and machine learning (Beygelzimer and Langford, 2009; Dud´ık, Langford, and Li, 2011; Kallus, 2018; Swaminathan and Joachims, 2015).”
Fair Decisions Despite Imperfect Predictions (2019, methodology): “We build on previous work on counterfactual inference and policy learning (Athey & Wager, 2017; Ensign et al., 2018; Gillen et al., 2018; Heidari & Krause, 2018; Joseph et al., 2016; Jung et al., 2018; Kallus, 2018; Kallus & Zhou, 2018; Lakkaraju & Rudin, 2017).”
Policy Evaluation with Latent Confounders via Optimal Balance (2019, methodology): “(7) over a class of kernel functions as defined by Definition 3, we can observe that the definition of JpW,µq looks very similar to the adversarial objective of Kallus [16], except that we have ⌫tpXi, Tiq terms instead of µtpXiq terms.”
Confounding-Robust Policy Improvement (2018, methodology): “This leads to strong generalization guarantees (Kallus 2017b, Kitagawa and Tetenov 2015, Wager and Athey 2017a) and can also allow one to incorporate domain-specific constraints that favor simple prescriptive decision policies that are interpretable, implementable, and/or satisfy operational…”
Accurate Inference for Adaptive Linear Models (2017, methodology): “, 2011; Swaminathan & Joachims, 2015) and policy learning (Athey & Wager, 2017; Kallus, 2017).”
Recursive Partitioning for Personalization using Observational Data (2016, background): “In learning from logged bandit feedback (Beygelzimer & Langford, 2009; Swaminathan & Joachims, 2015a;b; Kallus, 2017a), one is concerned with learning a good policy for a contextual multi-arm bandit problem based on logged data from another, known policy, rather than on-line interactions.”
Efficiently Breaking the Curse of Horizon: Double Reinforcement Learning in Infinite-Horizon Processes (2019, background): “…to further develop estimators that build on ours by leveraging MDP structure as we have here and perhaps combining this with ideas like balancing (Kallus, 2018), stability (Kallus and Uehara, 2019a), or blending (Thomas and Brunskill, 2016) that can improve the ﬁnite-sample performance in…”