Novelty and diversity in information retrieval evaluation

By C. Clarke, M. Kolla, Gordon V. Cormack, Olga Vechtomova, Azin Ashkan, Stefan Büttcher, Ian MacKinnon. Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 2008

1,088 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Information Systems

View Paper: https://www.semanticscholar.org/paper/15004aadabd967ac722a28a9c3bb39cf5bc32605

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

Citations in the corpus

Overview of the TREC-2014 Microblog Track (2014, methodology): “Alternative metrics we had considered include those based on gain [2] and the extension of mean average precision to graded relevance judgments [11].”
Ranking with Fairness Constraints (2017, background): “…w ij s to attempt to capture the amount of diversity item i would introduce at position k conditioned on the items that were placed at positions 1 , . . . , k − 1 (see [CG98, ZH08, CKC + 08, ZCL03, ZMKL05]), or casting it directly as an (unconstrained) multi-objective optimization problem [YS16].”
An axiomatic approach for result diversification (2009, background): “Result diversification lends itself as an effective solution to minimizing query abandonment [1, 9, 18].”
Information Retrieval Evaluation (2011, methodology): “Some of these definitions of relevance will require different or multiple metrics, such as graded relevance metrics like nDCG (see [46] for an example).”
Fairness of Exposure in Rankings (2018, background): “e accounts for uncertainty and diminishing returns from multiple relevant results [7, 25]. Under intrinsic diversity [24], the utility measure considers rankings as portfolios and reflects redundancy [10]. And under exploration diversity [24], the aim is to maximize utility to the user in the long t”
Diversity in Big Data: A Review (2017, neutral): “Novelty-based diversity measures deﬁne diversity with respect to the elements seen in the past (Refs.”
Diversifying search results (2009, background): “Several metrics have been proposed for evaluating the diversity of search results, including the %no metric [16], the k-call metric [6], and α-NDCG [7].”
Implicit Diversity in Image Summarization (2019, background): “Secondly, to ensure that the summary is representative of all relevant images, most prior work have used the idea of non-redundancy [14, 20, 53, 65, 70].”
Exploiting query reformulations for web search result diversification (2010, background): “The α-normalised discounted cumulative gain (α-NDCG [10]) metric balances relevance and diversity through the tuning parameter α.”
Towards Large-Scale Collaborative Planning: Answering High-Level Search Queries Using Human Computation (2011, result): “Finally, our work is related to research on diversifying search results (Santos, Macdonald, and Ounis 2010; Clarke et al. 2008; Agrawal et al. 2009; Gollapudi and Sharma 2009; Chen and Karger 2006) as a way to better answer faceted queries (e.g., “wedding planning”), which are queries that…”
A Survey on Context Learning (2017, result): “Empirical results show that this composite model leads to more robust diversification results and outperforms the baseline by 14 percent in terms of a-nDCG@20 [78].”
Diversity, Serendipity, Novelty, and Coverage (2016, methodology): “In addition to the traditional precision and recall metrics, the performance metrics included diversity (α-nDCG metric [Clarke et al. 2008]), novelty, and coverage.”
Expected reciprocal rank for graded relevance (2009, background): “This has led to various subtopic retrieval and diversity metrics [27, 9, 1].”
Search result diversification (2010, methodology): “[3], where novelty is viewed as the need to avoid redundancy, whereas diversity is viewed as the need to resolve ambiguity.”
Diversity in recommender systems - A survey (2017, methodology): “Clarke et al. [42] went a step further and tried to combine diversity and novelty into a new measure of retrieved document relevancy, which was based on Normalized Discounted Cumulative Gain measure.”
Document Comprehensiveness and User Preferences in Novelty Search Tasks (2015, positive): “Perhaps the more widely used measures in that category are α -nDCG [9] and ERR-IA [6].”
Information Retrieval for E-Discovery (2013, background): “[36] provides a good entry into the literature on search diversity and its evaluation, while Kekäläinen and Järvelin [79] is a foundational study in the use of graded relevance judgments in evaluation.”
Test Collection Based Evaluation of Information Retrieval Systems (2010, methodology): “described re-using existing test collections for diversity research: Clarke re-using a TREC QA collection [55]; Sanderson, an image search collection [221].”