Large-scale validation and analysis of interleaved search evaluation

By Olivier Chapelle, Thorsten Joachims, Filip Radlinski, Yisong Yue. 2012

238 citations recorded by the source corpus.

Subfields: Information Systems, Information Retrieval & Search

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Information Retrieval & Search

Citations in the corpus

Contextual Dueling Bandits (2015, methodology): “By presenting a mix or interleaving of two of the candidate rankings and observing the user’s response (Chapelle et al., 2012; Hofmann et al., 2013), it is possible for such a system to get feedback about user preferences.”
Counterfactual Estimation and Optimization of Click Metrics for Search Engines (2014, background): “More recently, interleaving has become an attractive technique to quickly identify the winner when two systems [8].”
Relative confidence sampling for efficient on-line ranker evaluation (2014, methodology): “Fortunately, on-line feedback can also be obtained using interleaved comparison methods , which give relative feedback about how one ranker compares to another and has been shown to be more reliable [6, 26].”
Generic Exploration and K-armed Voting Bandits (2013, unknown): “Further experiments were performed in (Chapelle et al., 2012).”
Exploring demographic information in social media for product recommendation (2015, methodology): “We adopt the balanced interleaving method [33] used for the evaluation of search engines to compare the performance of MARTold and B-MART.”
MergeDTS (2018, background): “RMED1 [31] extends an asymptotically optimal MAB algorithm, called Deterministic Minimum Empirical divergence (DMED) [11], by first proving an asymptotic lower bound on the cumulative regret of all dueling bandit algorithms, which has the order of Ω(K logT ), and pulling each pair of rankers the min”
Click Models for Web Search (2015, background): “, 2008] or interleaving [Chapelle et al., 2012], or assess the response time and resource requirements of a search system, we either need to have access to real users or somehow simulate them.”
Reducing Dueling Bandits to Cardinal Bandits (2014, methodology): “In practice, these two identical alternatives would be displayed as one, as would naturally happen in interleaved retrieval evaluation (Chapelle et al., 2012).”
Unbiased Learning-to-Rank with Biased Feedback (2016, methodology): “Similarly, randomized interleaving can detect preferences between different rankers reliably [2].”
The Netflix Recommender System (2015, background): “…choose for each Netﬂix session which algorithmic experience to offer, a design with better statistical performance for local metrics (e.g., see Chapelle et al. [2012]) but without the possibility of measuring changes to the overall engagement with the entire product or retention rates over…”
MergeRUCB: A Method for Large-Scale Online Ranker Evaluation (2015, methodology): “Interleaving methods have been successfully applied in large-scale settings [7, 8].”
Counterfactual Estimation and Optimization of Click Metrics in Search Engines: A Case Study (2015, methodology): “The more recent interleaving technique may also be used to quickly identify the winner between competing systems [6].”
Online Evaluation for Information Retrieval (2016, methodology): “Online evaluation in industry is often concerned with relatively incremental changes to a production system and may require hundreds of thousands or millions of impressions for detecting changes of the expected magnitude at the observed levels of variance [Chapelle et al., 2012].”
Reusing historical interaction data for faster online learning to rank for IR (2013, methodology): “An alternative is the Team Draft (TD) method, which addresses limitations of BI and proved reliable in large-scale IR evaluation [5, 21].”
Toward Predicting the Outcome of an A/B Experiment for Search Relevance (2015, methodology): “The standard way is to run an A/B experiment ( e.g. , [1, 16]) to directly measure online metrics, or to run an interleaving experiment that gives directional evaluation results [6, 24].”