Liberal relevance criteria of TREC -: counting on negligible documents?

By Eero Sormunen. 2002

202 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Education, Information Systems, Information Retrieval & Search, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Education

In Information Systems

In Information Retrieval & Search

In Evaluation & Benchmarks

Citations in the corpus

Pretrained Transformers for Text Ranking: BERT and Beyond (2020, methodology): “ B, regardless of the relevance judgments used for evaluation.39 This is a widely reproduced and robust ﬁnding, and these conclusions have been shown to hold across many different retrieval settings [Sormunen, 2002, Trotman and Jenkinson, 2007, Bailey et al., 2008, Wang et al., 2015]. This means tha”
Relevance assessment: are judges exchangeable and does it matter (2008, unknown): “Sormunen’s study reported re-judging work comparing TREC’s binary relevance scale with a four-level relevance scale [19], but system performance measures were not the concern here.”
The Notion of Relevance in Information Science: Everybody knows what relevance is. But, what is it really? (2016, methodology): “Ø Vakkari and Sormunen (2004) used 26 students to search 4 TREC-9 topics that already had pre-assigned relevance ratings by TREC assessors on a system that provided interactive relevance feedback capabilities, in order to study the consistency of user identification of relevant documents as pre-defi”
Methods for Evaluating Interactive Information Retrieval Systems with Users (2009, background): “, [245]), this has been the exception rather than the rule.”
On Fine-Grained Relevance Scales (2018, methodology): “, 805) have been reassessed in the study by Sormunen [19] using a 4-level scale.”
Developing a Test Collection for the Evaluation of Integrated Search (2010, methodology): “…65 17.7* (13 tasks with 5-10 terms) b) Work task context 65 35.7* c) Knowledge state 65 22.2* d) Ideal information 65 19.3* e) Search terms 65 9.4* (20 tasks with 3-6 terms) * minus stop words (using the 318 word list at ir.dcs.gla.ac.uk/resources/linguistic_utils/) The strength of the collection…”
Evaluation of contextual information retrieval effectiveness: overview of issues and research (2010, background): “Furthermore, in [98], an experimental study undertaken on TREC relevance judgements showed that 50% of assessed relevant documents were re-assessed as marginal relevant, 34% as fairly relevant and 16% as highly relevant.”
Information Retrieval Evaluation (2011, background): “The current judgments could be used as the starting point for other types of relevance judgments, such as removal of “duplicate” documents [24], or the use of graded relevance judgments [99, 162] or even the measurement of some type of learning effect (the TREC novelty track).”
Test Collection Based Evaluation of Information Retrieval Systems (2010, background): “Sormunen [240] re-examined TREC relevance judgments for 38 topics from TREC-7 and 8 using three relevance grades: not relevant, marginally relevant and highly relevant.”