Learning to Rank with Selection Bias in Personal Search

By Xuanhui Wang, Michael Bendersky, Donald Metzler, Marc Najork. Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 2016

301 citations recorded by the source corpus.

Click-through data has proven to be a critical resource for improving search ranking quality. Though a large amount of click data can be easily collected by search engines, various biases make it difficult to fully leverage this type of data. In the past, many click models have been proposed and successfully used to estimate the relevance for individual query-document pairs in the context of web search. These click models typically require a large quantity of clicks for each individual pair and this makes them difficult to apply in systems where click data is highly sparse due to personalized corpora and information needs, e.g., personal search. In this paper, we study the problem of how to leverage sparse click data in personal search and introduce a novel selection bias problem and address it in the learning-to-rank framework. This paper proposes a few bias estimation methods, including a novel query-dependent one that captures queries with similar results and can successfully deal with sparse data. We empirically demonstrate that learning-to-rank that accounts for query-dependent selection bias yields significant improvements in search effectiveness through online experiments with one of the world's largest personal search engines.

Subfields: Information Systems, Management Science and Operations Research

View Paper: https://www.semanticscholar.org/paper/b3ca9c2e302073264d2a3c2c2cee4d40b6fe908d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Management Science and Operations Research

Citations in the corpus

Unbiased Learning-to-Rank with Biased Feedback (2016, methodology): “Finally, our approach is similar in spirit to [28], where propensity-weighting is used to correct for selection bias when discarding queries without clicks during learning-torank.”
Situational Context for Ranking in Personal Search (2017, methodology): “In our ﬁnal ranking stage, we consider the position bias problem [26] similar to what is described in [44].”
Position Bias Estimation for Unbiased Learning to Rank in Personal Search (2018, methodology): “[31] study the selection bias at the query level when click-based training data is collected in learning-to-rank.”
Recommending what video to watch next: a multitask ranking system (2019, methodology): “Other approaches try to learn a bias term from position and apply it as a normalizer or regularizer [23, 34, 41].”
A General Framework for Counterfactual Learning-to-Rank (2018, methodology): “This approach is similar in spirit to [32], where propensity-weighting is used to correct for selection bias when discarding queries without clicks during learning-to-rank.”
TF-Ranking: Scalable TensorFlow Library for Learning-to-Rank (2018, methodology): “One proposed method [40, 41] is to compute Inverse Propensity Weights (IPW) for each position in the ranked list.”
Bias and Debias in Recommender System: A Survey and Future Directions (2020, background): “…leveraged counterfactual reasoning to eliminate the direct causal effect from exposure features on the prediction; Yang et al. [191] mitigates exposure bias through counterfactual samples; Liu et al. [108] disentangle the effect from exposure and preference with introducing information bottleneck.”
A Survey of Model Architectures in Information Retrieval (2025, unknown): “Additionally, some studies focus on unbiased relevance estimation using biased feedback (Wang et al., 2016; Joachims et al., 2017b,a; Ai et al., 2018c,b; Wang et al., 2018; Hu et al., 2019; Ren et al., 2022) while other focus on jointly optimizing effectiveness and fairness of the ranking systems…”