Why batch and user evaluations do not give the same results

By Andrew Turpin, William R. Hersh. 2001

200 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Information Systems, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jimmy J. Lin
2. Thorsten Joachims
3. Donna K. Harman
4. Lihong Li
5. Tobias Schnabel
6. John Riedl
7. William R. Hersh
8. Susan T. Dumais

In Information Systems

1. John Riedl
2. J. Konstan
3. Thorsten Joachims
4. Donna K. Harman
5. Diane Kelly
6. Lihong Li
7. William R. Hersh
8. Jimmy J. Lin

In Management Science and Operations Research

1. Thorsten Joachims
2. John Riedl
3. J. Konstan
4. Tobias Schnabel
5. Lihong Li
6. Donna K. Harman
7. Ankur Gupta
8. Shunbao Chen

Citations in the corpus

• Counterfactual Estimation and Optimization of Click Metrics for Search Engines (2014, background): “However, several authors have argued for several of its limitations ( e.g. , [3, 33]), in addition to the ones discussed in Sections 1 and 4.1; an alternative, “user-centered” evaluation emphasizes interaction between user and the search engine.”
• Test Collection Based Evaluation of Information Retrieval Systems (2010, unknown): “This experiment was repeated on another small collection with the same perhaps surprising conclusion [258].”
• A comparison of user and system query performance predictions (2010, background): “However, there has been related work examining and perhaps challenging the relationship between system-centered retrieval effectiveness measures and actual user performance in traditional IR settings [1, 16, 28, 30].”
• Assessor Differences and User Preferences in Tweet Timeline Generation (2015, background): “Early studies suggest that “better” systems (as measured by system-oriented metrics) don’t necessarily translate into better task performance [13, 29, 30].”
• Toward Predicting the Outcome of an A/B Experiment for Search Relevance (2015, background): “However, the non-interactive nature also causes limitations, as pointed out by several authors [2, 28], as they are approximations of the quality of end user experience.”
• Information Retrieval: A Biomedical and Health Perspective (2020, background): “Gotzsche and Olsen [332] found that subsequent references to seven RCTs of mammography screening tended to omit important limitations of the trials.”
• Evaluating collaborative filtering recommender systems (2004, unknown): “Turpin and Hersh’s [2001] study of search engines provides support for this position.”
• Short-Term Satisfaction and Long-Term Coverage: Understanding How Users Tolerate Algorithmic Exploration (2018, background): “The problem with rating-based studies is that they provide no insight into how and whether recommendations would impact users in their tasks [41], a shortcoming that is addressed by task-based studies.”
• Evaluation of contextual information retrieval effectiveness: overview of issues and research (2010, unknown): “…settings; for instance, experiments in the interactive track of TREC have shown that significant differences in mean average precision in a batch evaluation did not correlate with interactive user performance for a small number of topics in the instance recall and question answering tasks [111].”
• Some(what) grand challenges for information retrieval (2008, background): “…generally not statistically significant (e.g. Spärck Jones, 2005); and thirdly, that such improvement, as determined in TREC-style evaluation, rarely, if ever, leads to improved performance by human searchers in interactive IR systems (e.g. Turpin & Hersh, 2001; Turpin & Scholer, 2006).”
• Information Retrieval Evaluation (2011, background): “Bill Hersh from Oregon Health Sciences Institute ran the genomics track for retrieval from medical information starting in TREC 2004.”
• Research-paper recommender systems: a literature survey (2015, background): “More precisely, several researchers have shown that results from offline evaluations do not necessarily correlate with results from user studies or online evaluations [93], [270], [271], [279–282].”
• User performance versus precision measures for simple search tasks (2006, background): “The relationship between system effectiveness and user performance for a question answering task was considered by Turpin and Hersh [17].”
• Is seeing believing?: how recommender system interfaces affect users' opinions (2003, background): “Turpin and Hersh found that users of two search engines, one much better than the other, showed equal satisfaction with the systems [17], suggesting that users are not very sensitive to differences in search engine quality.”