User performance versus precision measures for simple search tasks

By Andrew Turpin, Falk Scholer. Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 2006

444 citations recorded by the source corpus.

Subfields: Information Systems

View Paper: https://www.semanticscholar.org/paper/84fcd3ce562b810b18a0dedb62b6144876939f97

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

Citations in the corpus

How does clickthrough data reflect retrieval quality? (2008, background): “of the metrics have been shown to not necessarily correlate with user-centric performance measures [22].”
Alternatives to Bpref (2007, background): “Finally, it should be noted that we have not examined IR metrics from the viewpoint of user satisfaction: It remains to be seen, for example, how recent criticisms of AveP [14] extend to different experimental environments and different metrics.”
Methods for Evaluating Interactive Information Retrieval Systems with Users (2009, background): “There have been a series of studies that have compared results of batch-mode and user studies and found that systems which perform better in batch-mode studies do not always do so in user studies [129, 243, 275, 276].”
Some(what) grand challenges for information retrieval (2008, background): “First, it is clearly the case that IR as practiced is inherently interactive; secondly, it is clearly the case that the new models and associated representation and ranking techniques lead to only incremental (if that) improvement in performance over previous models and techniques, which is gener”
Test Collection Based Evaluation of Information Retrieval Systems (2010, methodology): “[260] A. Turpin and F. Scholer, “User performance versus precision measures for simple search tasks,” in Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 11–18, New York, NY, USA: ACM, 2006.”
A comparison of user and system query performance predictions (2010, background): “However, there has been related work examining and perhaps challenging the relationship between system-centered retrieval eﬀectiveness measures and actual user performance in traditional IR settings [1, 16, 28, 30].”
Wide and Deep Convolutional Neural Networks for Electricity-Theft Detection to Secure Smart Grids (2018, methodology): “3) Performance Metrics: In this paper, we conduct the experiments by considering two performance metrics: area under curve (AUC) [40] and mean average precision (MAP) [41].”
Assessor Differences and User Preferences in Tweet Timeline Generation (2015, background): “Unlike some previous work that artiﬁcially manipulated system output to generate results of a particular level of eﬀectiveness (e.g., [4, 30]), our procedure yields more realistic comparisons.”
Beyond position bias: examining result attractiveness as a source of presentation bias in clickthrough data (2010, background): “Additionally, some metrics based on human judgments have been shown to not necessarily correlate with more user-centric performance measures [ 22 ].”
TOPSIG: topology preserving document signatures (2011, methodology): “Turpin and Scholer [19] performed retrieval experiments where users completed search tasks using search results with MAP scores between 55% and 95%.”
Learning diverse rankings with multi-armed bandits (2008, background): “As training data is being collected, it immediately impacts the rankings shown.”
A Theoretical Analysis of NDCG Type Ranking Measures (2013, background): “There seem to be many ways to deﬁne ranking measures and several evaluation measures have been proposed [11, 32, 4, 1, 28].”
Deep Reinforcement Learning for List-wise Recommendations (2017, methodology): “To evaluate the performance of the proposed framework, we select MAP [27] and NDCG [6] as the metrics to measure the performance.”
Sequential Projection Learning for Hashing with Compact Codes (2010, methodology): “For Hamming ranking based evaluation, Mean Average Precision (MAP) is computed which approximates the area under precision-recall curve (Turpin & Scholer, 2006).”
Learning Visually-Grounded Semantics from Contrastive Adversarial Samples (2018, background): “, both positive and negative objects, P (k) is the precision at cut-off k in the list, positive(k) is an indicator function equaling 1 if the object at rank k is a positive one, 0 otherwise (Turpin and Scholer, 2006).”
Deep reinforcement learning for page-wise recommendations (2018, methodology): “For offline test, we select Precision@20, Recall@20, F1-score@20 [9], NDCG@20 [11] and MAP [33], as the metrics to measure the performance.”