Position Bias Estimation for Unbiased Learning to Rank in Personal Search

By Xuanhui Wang, Nadav Golbandi, Michael Bendersky, Donald Metzler, Marc Najork. 2018

301 citations recorded by the source corpus.

A well-known challenge in learning from click data is its inherent bias and most notably position bias. Traditional click models aim to extract the ‹query, document› relevance and the estimated bias is usually discarded after relevance is extracted. In contrast, the most recent work on unbiased learning-to-rank can effectively leverage the bias and thus focuses on estimating bias rather than relevance [20, 31]. Existing approaches use search result randomization over a small percentage of production traffic to estimate the position bias. This is not desired because result randomization can negatively impact users' search experience. In this paper, we compare different schemes for result randomization (i.e., RandTopN and RandPair) and show their negative effect in personal search. Then we study how to infer such bias from regular click data without relying on randomization. We propose a regression-based Expectation-Maximization (EM) algorithm that is based on a position bias click model and that can handle highly sparse clicks in personal search. We evaluate our EM algorithm and the extracted bias in the learning-to-rank setting. Our results show that it is promising to extract position bias from regular clicks without result randomization. The extracted bias can improve the learning-to-rank algorithms significantly. In addition, we compare the pointwise and pairwise learning-to-rank models. Our results show that pairwise models are more effective in leveraging the estimated bias.

Subfields: Artificial Intelligence, Computer Networks and Communications, Information Systems, Information Retrieval & Search

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Information Systems

In Information Retrieval & Search

Citations in the corpus

Judging LLM-as-a-judge with MT-Bench and Chatbot Arena (2023, background): “This bias is not unique to our context and has been seen in human decision-making [3, 29] and other ML domains [18, 35].”
TF-Ranking: Scalable TensorFlow Library for Learning-to-Rank (2018, methodology): “One proposed method [40, 41] is to compute Inverse Propensity Weights (IPW) for each position in the ranked list.”
Bias and Debias in Recommender System: A Survey and Future Directions (2020, background): “Propensity Score Weight the data Theoretical-unbiasedness High variance, Requiring proper propensities, Fail to model trust [6, 67, 77, 137, 147, 156, 172] [9, 52, 77, 77, 135, 160, 174] [8, 34, 60]”
A General Framework for Counterfactual Learning-to-Rank (2018, methodology): “Practical approaches to estimating the propensities are given in [1, 10, 15, 33].”
Deep Learning for Anomaly Detection (2020, background): “(iii) This approach may build upon well-established ranking techniques and theories from areas like learning to rank [85, 87, 158].”
JudgeLM: Fine-tuned Large Language Models are Scalable Judges (2023, unknown): “Position bias means that the LLM judges prefer answers in a certain position and it widely exists in natural language processing tasks (Ko et al., 2020; Wang et al., 2018) and decision-making of humans (Blunch, 1984; Raghubir & Valenzuela, 2006).”
Unbiased Learning-to-Rank with Biased Feedback (2016, methodology): “Since the original paper appeared, another technique for propensity estimation from observational data has been proposed that could also be used [Wang et al., 2018].”
Demographics Should Not Be the Reason of Toxicity: Mitigating Discrimination in Text Classifications with Instance Weighting (2020, methodology): “For example, the Inverse Propensity Score (IPS) (Rosen-baum and Rubin, 1983) method has been successfully applied for causal effect analyses (Austin and Stuart, 2015), selection bias (Schonlau et al., 2009), position bias (Wang et al., 2018; Joachims et al., 2017) and so on.”
A Survey of Model Architectures in Information Retrieval (2025, unknown): “…some studies focus on unbiased relevance estimation using biased feedback (Wang et al., 2016; Joachims et al., 2017b,a; Ai et al., 2018c,b; Wang et al., 2018; Hu et al., 2019; Ren et al., 2022) while other focus on jointly optimizing effectiveness and fairness of the ranking systems…”