Counterfactual Estimation and Optimization of Click Metrics in Search Engines: A Case Study
By Lihong Li, Shunbao Chen, Jim Kleban, Ankur Gupta. 2015
96 citations recorded by the source corpus.
Subfields: Information Systems, Management Science and Operations Research, Marketing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Information Systems
- Suchi Saria
- Stefan Riezler
- Artem Sokolov
- Manik Varma
- Yashoteja Prabhu
- Fernando Diaz
- Yi-Hsun Su
- Thorsten Joachims
In Management Science and Operations Research
- Stefan Riezler
- Carolin (Haas) Lawrence
- Aleksandrs Slivkins
- Artem Sokolov
- Yi-Hsun Su
- Suchi Saria
- Noveen Sachdeva
- Adarsh Subbaswamy
In Marketing
- Fernando Diaz
- Manik Varma
- Thorsten Joachims
- Mounia Lalmas
- Suchi Saria
- Himanshu Jain
- Aleksandrs Slivkins
- Yashoteja Prabhu
Citations in the corpus
- Off-policy Bandits with Deficient Support (2020, methodology): “Such contextual-bandit data is logged in large quantities as a by-product of normal system operation [12, 17, 18], making it an attractive and low-cost source of training data.”
- Learning Treatment-Response Models from Multivariate Longitudinal Data (2017, background): “…we can leverage observational data about visit and click patterns to measure user responses to interventions (e.g., a new ad campaign or a specific change to the website), allowing us to rapidly improve their web experiences; e.g., Bottou et al. (2013); Li et al. (2015); Schnabel et al. (2016).”
- Learning to Rank with Selection Bias in Personal Search (2016, unknown): “in prior work [24, 25].”
- Counterfactual Learning from Bandit Feedback under Deterministic Logging : A Case Study in Statistical Machine Translation (2017, background): “…in various fields that deal with partial feedback, namely contextual bandits (Langford et al. (2008); Strehl et al. (2010); Dudik et al. (2011); Li et al. (2015), inter alia ), reinforcement learning (Sutton and Barto (1998); Precup et al. (2000); Jiang and Li (2016); Thomas and Brunskill…”
- Lessons from Real-World Reinforcement Learning in a Customer Support Bot (2019, methodology): “Similar ideas have been proposed in (Li et al., 2015).”
- Extreme Multi-label Loss Functions for Recommendation, Tagging, Ranking & Other Missing Label Applications (2016, background): “Propensities have also been used for off policy evaluation, whereby feedback data from the interaction logs of an existing system is used to evaluate a new system [7, 23, 24, 40, 41].”
- Towards a Fair Marketplace: Counterfactual Evaluation of the trade-off between Relevance, Fairness & Satisfaction in Recommendation Systems (2018, methodology): “Off-policy evaluation, and counterfactual estimation of metrics has recently gained a lot of attention in the research community, with recent work focusing on counterfactual estimation of search engine metrics [14], for evaluating slate recommendations [27].”
- Introduction to Multi-Armed Bandits (2019, background): “Decision Service, a system for large-scale applications of contextual bandits, is available at GitHub.3 Bietti et al. (2018) compare the empirical performance of various contextual bandit algorithms. Li et al. (2015) provide a case study of policy optimization and training in web search engine optim”
- MultiVerse: Causal Reasoning using Importance Sampling in Probabilistic Programming (2019, background): “, 2019; Oberst and Sontag, 2019), advertisement and search (Bottou et al., 2013; Swaminathan and Joachims, 2015; Li et al., 2015; Gilotte et al., 2018), translation (Lawrence et al.”