Exploration scavenging

By J. Langford, Alexander L. Strehl, Jennifer Wortman Vaughan. 2008

137 citations recorded by the source corpus.

Subfields: Computer Science Applications, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Science Applications

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Can Neural Machine Translation be Improved with User Feedback? (2018, background): “To our surprise, the bandit-to-supervised conversion proved to be very hard to beat, despite theoretical indications of poor generalization for exploration-free learning from logged data (Langford et al., 2008; Strehl et al., 2010).”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, neutral): “In contrast to the other applications discussed, policy evaluation for recommender systems is typically formulated within a contextual bandit problem (Langford et al., 2008; Li et al., 2010), where states may correspond to user history and actions correspond to recommendations.”
Doubly Robust Policy Evaluation and Optimization (2014, background): “Contextual bandits (Auer et al., 2002/03; Lang-ford and Zhang, 2008), sometimes known as asso-Miroslav Dud´ık is Senior Researcher and John Langford is Principal Researcher, Microsoft Research, New York, New York, USA (e-mail: mdudik@microsoft.com; jcl@microsoft.com).”
Counterfactual Estimation and Optimization of Click Metrics for Search Engines (2014, methodology): “Such counterfactual methods have shown promise in a few important Web applications recently like advertising [5, 7, 19, 20, 30], content recommendation [22].”
Doubly Robust Policy Evaluation and Learning (2011, background): “Here, we focus on the offline version: we assume access to historic data, but no ability to gather new data (Langford et al., 2008; Strehl et al., 2011).”
Learning from Logged Implicit Exploration Data (2010, background): “The main restrictive assumption adopted by the Ex-ploration Scavenging paper (Langford et al., 2008) is that the logging policy chooses actions independently of the input.”
Off-policy evaluation for slate recommendation (2016, background): “This assumption is also required for IPS, otherwise unbiased off-policy evaluation is impossible [24].”
Unbiased offline evaluation of contextual-bandit-based news article recommendation algorithms (2010, methodology): “This error bound improves a previous result [17, Theorem 5] for a similar oﬄine evaluation algorithm and similarly provides a sharpened analysis for the T = 1 special case for policy evaluation in reinforcement learning [15].”
Unbiased Learning-to-Rank with Biased Feedback (2016, methodology): “Unlike the uniform ranking randomization of [28] (with its high performance impact) or swapping adjacent pairs as in [5], we swap documents in different ranks to the top position randomly as in [12].”
Fair Decisions Despite Imperfect Predictions (2019, methodology): “Finally, our work also overlaps with previous work on contextual bandits (Dudík et al., 2011; Langford et al., 2008; Agarwal et al., 2014) and reinforcement learning (Jabbari et al., 2016; Sutton & Barto, 1998).”
Sample-efficient Nonstationary Policy Evaluation for Contextual Bandits (2012, background): “The IPS evaluator can take advantage of scavenged exploration through replacing p by an estimator ˆ p [16, 24], but it does not allow evaluation of nonstationary policies and does not take advantage of a reward estimator.”
Counterfactual Estimation and Optimization of Click Metrics in Search Engines: A Case Study (2015, methodology): “Such counterfactual methods have shown promise in a few important Web applications recently [4, 5, 17, 18, 21, 29, 30, 25].”
Online Evaluation for Information Retrieval (2016, methodology): “In fact, in many situations it is known to be impossible [Langford et al., 2008].”
Toward Predicting the Outcome of an A/B Experiment for Search Relevance (2015, background): “As explained in Section 3, the approach we take here is closely related to previous oﬄine evaluation work for recommender and advertising systems based on contextual ban-dits [18, 21], and to causal inference in statistics [17] as well.”
The Self-Normalized Estimator for Counterfactual Learning (2015, background): “Lower bounds for domain adaptation [8] and impossibility results for off-policy evaluation [9], hence, also apply to propensity score matching [7], costing [10] and other importance sampling approaches to BLBF.”
Toward Minimax Off-policy Value Estimation (2015, methodology): “Off-policy evaluation has been found useful in several important applications (Langford et al., 2008, Li et al., 2011, Bottou et al., 2013) and can also be regarded as a key building block for policy optimization which, as in supervised learning, can often be reduced to evaluation, as long as the…”
An unbiased offline evaluation of contextual bandit algorithms with generalized linear models (2011, methodology): “Furthermore, if A is a stationary policy that does not change over trials, data may be used more efficiently via propensity scoring (Langford et al., 2008; Strehl et al., 2011) and related techniques like doubly robust estimation (Dudı́k et al.”
Reusing historical interaction data for faster online learning to rank for IR (2013, background): “In particular, solutions have been proposed in the areas of news recommendation [1, 16] and ad placement [15, 24].”
EFFICIENT ABSTRACTION SELECTION IN REINFORCEMENT LEARNING (2013, background): “Contextual bandit problems are a useful way to model many real-world tasks, for example, selecting ads to place alongside web pages (Langford and Zhang 2007; Langford et al. 2008).”