Efficient Policy Learning

By Susan Athey, Stefan Wager. 2017

185 citations recorded by the source corpus.

We consider the problem of using observational data to learn treatment assignment policies that satisfy certain constraints specified by a practitioner, such as budget, fairness, or functional form constraints. This problem has previously been studied in economics, statistics, and computer science, and several regret-consistent methods have been proposed. However, several key analytical components are missing, including a characterization of optimal methods for policy learning, and sharp bounds for minimax regret. In this paper, we derive lower bounds for the minimax regret of policy learning under constraints, and propose a method that attains this bound asymptotically up to a constant factor. Whenever the class of policies under consideration has a bounded Vapnik-Chervonenkis dimension, we show that the problem of minimax-regret policy learning can be asymptotically reduced to first efficiently evaluating how much each candidate policy improves over a randomized baseline, and then maximizing this value estimate. Our analysis relies on uniform generalizations of classical semiparametric efficiency results for average treatment effect estimation, paired with sharp concentration bounds for weighted empirical risk minimization that may be of independent interest.

Subfields: Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Balanced Linear Contextual Bandits (2018, positive): “There is also a signiﬁcant body of research that leverages balancing for ofﬂine evaluation and learning of contextual bandit or reinforcement learning policies from logged data (Strehl et al. 2010), (Dud ´ ık, Langford, and Li 2011), (Li et al. 2012), (Dud ´ ık et al. 2014), (Li et al. 2014), (Swami”
Estimation Considerations in Contextual Bandits (2017, methodology): “[10]) derives efficient methods for offline policy estimation when the policy is constrained to be of limited complexity.”
Balanced Policy Evaluation and Learning (2017, methodology): “For policy learning, [1, 13] minimize this estimator via weighted classification.”
Learning When-to-Treat Policies (2019, methodology): “4The idea of cross-fitting has gained growing popularity recently to reduce the effect of own-observation bias and enable results on semiparametric rates of convergence using generic nuisance component estimates (Athey and Wager, 2017; Chernozhukov et al., 2016a; Schick, 1986).”
Quasi-oracle estimation of heterogeneous treatment effects (2017, methodology): “, 2011), and is also a key component of several proposals for learning decision rules (Athey and Wager, 2017; Hirano and Porter, 2009).”
Confounding-Robust Policy Improvement (2018, methodology): “…well as approaches to predict individual-level causal effects from observational data (Künzel et al. 2017, Nie and Wager 2017, Shalit et al. 2017, Wager and Athey 2017b), operate under the controversial assumption of unconfoundedness, which requires that the data are sufficiently informative…”
Accurate Inference for Adaptive Linear Models (2017, methodology): “, 2011, Swaminathan and Joachims, 2015] and policy learning [Athey and Wager, 2017, Kallus, 2017]. These focus on efficiently estimating the reward (or value) of a certain test policy using data collected from a different policy. For linear models, this reduces to accurately estimating the predictio”
Augmented minimax linear estimation (2017, methodology): “Under exogeneity assumptions, this estimand can be used to compare policies for assigning personalized treatments, and estimators for it form a key building block in methods for estimation of optimal treatment policies (Athey and Wager, 2017).”
Constrained Bayesian Optimization with Noisy Experiments (2017, background): “This work is related to policy optimization (Athey and Wager, 2017), which seeks to learn an optimal mapping from context to action.”
Faster Rates for Policy Learning (2017, background): “to [1] for a recent overview of the literature.”
Stratification Trees for Adaptive Randomization in Randomized Controlled Trials (2018, background): “…for the Quest high performance computing facility at Northwestern University. work of Breiman (see Breiman et al., 1984; Gyorfi et al., 1996, for classical textbook treatments), and has seen a recent resurgence in econometrics (examples include Athey and Imbens, 2016; Athey and Wager, 2017).”
Targeted Undersmoothing. (2017, background): “18See [8] and [50] for interesting approaches to estimating and performing inference for optimal treatment strategies.”
Fair Decisions Despite Imperfect Predictions (2019, methodology): “We build on previous work on counterfactual inference and policy learning (Athey & Wager, 2017; Ensign et al., 2018; Gillen et al., 2018; Heidari & Krause, 2018; Joseph et al., 2016; Jung et al., 2018; Kallus, 2018; Kallus & Zhou, 2018; Lakkaraju & Rudin, 2017).”
Statistically Efficient Off-Policy Policy Gradients (2020, unknown): “, Athey and Wager (2017); Kallus (2018); Kallus and Zhou (2018); Swaminathan and Joachims (2015) for H = 0 and Chen et al.”