Relevance assessments and retrieval system evaluation

By Michael E. Lesk, Gerard A. Salton. 1968

Subfields: Information Retrieval & Search, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Retrieval & Search

In Evaluation & Benchmarks

Citations in the corpus

The TREC question answering track (2001, methodology): “Following the document relevance judgment literature (Lesk and Salton, 1969), we can compute the overlap in the sets of strings that were judged correct.”
Measuring Retrieval Effectiveness Based on User Preference of Documents (1995, background): “(10) Two rankings agree on a pair of documents d, d′ ∈ D if both of them rank d and d′ in the same order, i.e., ≻1({d, d ′}) = ≻2({d, d ′}).”
Relevance assessment: are judges exchangeable and does it matter (2008, background): “Lesk and Salton found that de- SIGIR 08, July 20 24, 2008, Singapore.”
How reliable are the results of large-scale information retrieval experiments? (1998, background): “It is well known that the reliability of measurement of a system depends on the quality of the relevance judgements, and that relevance assessors are rarely in exact agreement [4, 5, 7, 11, 14, 15].”
Indexing by Latent Semantic Analysis (1990, unknown): “Get rid of this junk [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] [22] [23] [24] [25] [26] [27] [28] [29] [30] [31] [32] [33] [34] [ 35 ] [36] [37] [38]”
Information Retrieval Evaluation (2011, background): “These issues had plagued the Cranfield experiments and a new test collection, the ISPRA collection, was built specifically to address this problem [117, 146].”
Variations in relevance judgments and the measurement of retrieval effectiveness (1998, methodology): “Previous studies have used the overlap of the relevant document sets to quantify the amount of agreement among di erent sets of relevance assessments [9].”
The Notion of Relevance in Information Science: Everybody knows what relevance is. But, what is it really? (2016, background): “Most of them are modeled on the first and often cited Lesk and Salton (1968) study that had actual data on the complaint voiced by critics.”
Test Collection Based Evaluation of Information Retrieval Systems (2010, methodology): “[164] M. E. Lesk and G. Salton, “Relevance assessments and retrieval system evaluation*1,” Information Storage and Retrieval, vol. 4, no. 4, pp. 343–359, doi:10.1016/0020-0271(68)90029-6, 1968.”
The TREC-8 Question Answering Track Evaluation (2000, methodology): “Following the document relevance judgment literature [3], we can compute the overlap in the sets of strings that were judged correct.”
Information Retrieval: A Biomedical and Health Perspective (2020, background): “Other have noted, however, that much more must be done to maintain trust in scientific research, including addressing systematically some of the flaws that will be described later in this chapter [126].”
Building a question answering test collection (2000, methodology): “Following the document relevance judgment literature [9], we can compute the overlap in the sets of strings that were judged correct.”