The Self-Normalized Estimator for Counterfactual Learning

By Adith Swaminathan, Thorsten Joachims. 2015

435 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

Citations in the corpus

Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback (2018, methodology): “Our contribution is to modify the self-normalizing estimator (Kong, 1992; Precup et al., 2000; Swaminathan and Joachims, 2015b; Joachims et al., 2018) to be applicable to neural networks.”
Can Neural Machine Translation be Improved with User Feedback? (2018, methodology): “For the special case of deterministically logged rewards Lawrence et al. (2017b) introduced the Deterministic Propensity Matching (DPM) objective with self-normalization as a multiplicative control variate (Swaminathan and Joachims, 2015)”
Recommendations as Treatments: Debiasing Learning and Evaluation (2016, methodology): “Applied to the IPS estimator, we know that This yields the Self-Normalized Inverse Propensity Scoring (SNIPS) estimator (Trotter & Tukey, 1956; Swaminathan & Joachims, 2015) The SNIPS estimator often has lower variance than the IPS estimator but has a small bias (Hesterberg, 1995).”
Deep-Treat: Learning Optimal Personalized Treatments From Observational Data Using Neural Networks (2018, methodology): “Self normalizing estimator, a variant of IPS estimator, is also used to learn counterfactuals (Swaminathan and Joachims 2015b).”
Balanced Policy Evaluation and Learning (2017, methodology): “A refinement, SNPOEM [43] uses a normalized and clipped IPW (NCIPW) estimator (and regularizes variance): ⌧̂M -NCIPW(⇡) = ⌧̂WM -NCIPW(⇡), W M -NCIPW i (⇡) = W M -CIPW i (⇡)/ Pn i0=1 W M -CIPW i0 (⇡).”
Estimating Counterfactual Treatment Outcomes over Time Through Adversarially Balanced Representations (2020, background): “Learning optimal policies: A related problem to ours involves learning the optimal treatment policies from logged data (Swaminathan & Joachims, 2015a;b; Atan et al., 2018).”
Off-policy Bandits with Deficient Support (2020, methodology): “However, policy learning algorithms based on IPS and related estimators [20, 21, 24–26] require the assumption that the logging policy has full support for every policy in the policy space.”
Intrinsically Efficient, Stable, and Bounded Off-Policy Evaluation for Reinforcement Learning (2019, methodology): “Therefore self-normalized IS (SNIS; aka Hájek) is often used [28], which divides IS by the average of density ratios.”
Offline A/B Testing for Recommender Systems (2018, methodology): “alised importance sampling. We know that Eπ p [W]= 1. Using the empirical average 1 n ˝ (x,a,r)∈S n w(a,x)as a global ratio control variate, we have the normalized importance sampling (NIS) estimator [21, 24]: RˆNIS(πt)= 1 ˝ (x,a,r)∈S n w(a,x) Õ (x,a,r)∈S n w(a,x)r The normalizing constant is equal ”
Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processes (2019, background): “A common variant of IS is the self-normalized estimate ∑T t=0 En[λ̂trt] En[λ̂t] (Swaminathan and Joachims, 2015), which trades off some bias for variance but does not make IS efficient.”
Policy Evaluation and Optimization with Continuous Treatments (2018, methodology): “As discussed in [22], IPW methods can suffer from variance in estimates due to the propensity weights.”
A Contextual Bandit Bake-off (2018, neutral): “Although some techniques attempt to remove a dependence on such encoding choices through appropriately designed counterfactual loss estimators (Dudik et al., 2011b; Swaminathan and Joachims, 2015), these may be imperfect in practice, and particularly in an online scenario.”
Counterfactual Evaluation and Learning for Search, Recommendation and Ad Placement (2016, background): “…(also called estimation of treatment eﬀects and oﬀ-policy evaluation) and counterfactual learning (also called policy optimization) has made rapid progress in recent years [8, 3, 7, 6, 2, 13, 22, 9, 11, 14, 15, 19, 24, 12, 18], and it is starting to have impact in commercial applications.”
Counterfactual Learning for Machine Translation: Degeneracies and Solutions (2017, background): “The new objective is the reweighted deterministic propensity matching (DPM+R) objective: recovers IPS with reweighting, ˆ V IPS+R ( π w ) (Swaminathan and Joachims, 2015).”
Confounding-Robust Policy Improvement (2018, methodology): “This generalizes the popular class of IPWbased approaches to policy learning, which optimize an unbiased estimator for policy value under unconfoundedness [15, 28, 27].”
Efficient Counterfactual Learning from Bandit Feedback (2018, methodology): “to use batch data collected by a logging policy to estimate the value of a counterfactual policy or algorithm without employing it (Li et al. 2010; Strehl et al. 2010; Li et al. 2011; Li et al. 2012; Swaminathan and Joachims 2015a; Swaminathan and Joachims 2015b; Wang, Agarwal, and Dudik 2017; Swami”
Data-Driven Market-Making via Model-Free Learning (2020, methodology): “…use any of the three main off-policy evaluation methods: direct method [Bertsekas et al. , 1995; Lagoudakis and Parr, 2003; Sutton and Barto, 2018], importance sampling [Swaminathan and Joachims, 2015], and doubly robust method [Dud´ık et al. , 2014; Jiang and Li, 2015; Robins et al. , 1994].”