How does clickthrough data reflect retrieval quality?

By Filip Radlinski, Madhu Kurup, Thorsten Joachims. 2008

356 citations recorded by the source corpus.

Automatically judging the quality of retrieval functions based on observable user behavior holds promise for making retrieval evaluation faster, cheaper, and more user centered. However, the relationship between observable user behavior and retrieval quality is not yet fully understood. We present a sequence of studies investigating this relationship for an operational search engine on the arXiv.org e-print archive. We find that none of the eight absolute usage metrics we explore (e.g., number of clicks, frequency of query reformulations, abandonment) reliably reflect retrieval quality for the sample sizes we consider. However, we find that paired experiment designs adapted from sensory analysis produce accurate and reliable statements about the relative quality of two retrieval functions. In particular, we investigate two paired comparison tests that analyze clickthrough data from an interleaved presentation of ranking pairs, and we find that both give accurate and consistent results. We conclude that both paired comparison tests give substantially more accurate and sensitive evaluation results than absolute usage metrics in our domain.

Subfields: Artificial Intelligence, Computer Science Applications, Information Systems, Information Retrieval & Search, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Information Systems

In Information Retrieval & Search

In Evaluation & Benchmarks

Citations in the corpus

Contextual Dueling Bandits (2015, background): “This problem naturally arises in fields such as information retrieval (IR) and recommender systems, where it has been shown that user feedback is considerably more reliable when interpreted as relative comparisons rather than absolute labels (Radlinski et al., 2008).”
Coactive Learning (2015, methodology): “Depending on the feedback, we inferred which of the two rankings was preferred using the Team-Game method proposed by Radlinski et al. (2008).”
Up next: retrieval methods for large scale related video suggestion (2014, background): “It has been shown that implicit user feedback is often influenced by presentation bias [32], and click metrics do not fully correlate with relevance [21].”
Relative Upper Confidence Bound for the K-Armed Dueling Bandit Problem (2013, unknown): “One effective way to achieve this is to use interleaved comparisons (Radlinski et al., 2008), which interleave the documents proposed by two different rankers and presents the resulting list to the user, whose resulting click feedback is used to infer a noisy preference for one of the rankers.”
Click Models for Web Search (2015, background): “robust estimations for quality metrics such as abandonment rate or mean reciprocal rank (MRR) [Radlinski et al., 2008].”
A probabilistic method for inferring preferences from clicks (2011, background): “Categories and Subject Descriptors H.3 [Information Storage and Retrieval]: H.3.3 Information Search and Retrieval General Terms Algorithms, Experimentation Keywords Evaluation, Implicit feedback, Interleaved comparison 1.”
Maximum Selection and Ranking under Noisy Comparisons (2017, background): “Based on these small random comparisons, the company would like to rank the ads according to their appeal Radlinski & Joachims [2007], Radlinski et al. [2008].”
Interactively optimizing information retrieval systems as a dueling bandits problem (2009, methodology): “In the search example, P(w w0) refers to the fraction of users who prefer the results produced by w over those of w0. One can regard (w,w0) as the distinguishability between w and w0. Algorithms learn only via observing comparison results (e.g., from interleaving ( Radlinski et al., 2008b )).”
Beat the Mean Bandit (2011, methodology): “However, for search engines there exist reliable methods for inferring preference feedback (e.g. is A better than B) from clicks (Radlinski et al., 2008).”
The FacT: Taming Latent Factor Models for Explainability with Factorization Trees (2019, methodology): “3 To increase the sensitivity of comparison between two recommendation algorithms, we conduct interleaved test [19] in this cold-start study.”
MergeRUCB: A Method for Large-Scale Online Ranker Evaluation (2015, methodology): “A common approach is to use interleaved comparison meth-ods [7, 8, 13, 15, 18, 23, 25], in which the document lists proposed by two candidate rankers for a given query are interleaved and the resulting list presented to the user, whose clicks are used to infer a noisy preference for one ranker over…”
Refining Recency Search Results with User Click Feedback (2011, result): “…evaluating our online re-ranking function based on CTR@1, we also compare our results with following other click metrics for ranking proposed in [26]: • Query CTR is the average number of clicks for each query Thus, for all metrics, higher values are assumed to indicate better ranking qualities.”
Reusing historical interaction data for faster online learning to rank for IR (2013, methodology): “Previously, DBGD was evaluated with a method called Balanced Interleave (BI) [10, 14, 21].”
Expected reciprocal rank for graded relevance (2009, methodology): “Leveraging the conclusions from a previous study [19, table 2], we experimented with the most discriminative metrics according to the ﬁndings of that paper, namely: QCTR Number of clicks in a session.”
Online Evaluation for Information Retrieval (2016, background): “In particular, online evaluation addresses questions about users’ experience with an IR system that are quite distinct from those answered by offline evaluation using a test-collection-based approach, surveyed by Sanderson [2010]. Test-collection-based evaluation models users at varying levels of ab”
Copeland Dueling Bandits (2015, methodology): “[33] F.”