Do batch and user evaluations give the same results?

By William R. Hersh, Andrew Turpin, Susan Price, Benjamin Chan, Dale Kraemer, Lynetta Sacherek, Daniel Olson. 2000

160 citations recorded by the source corpus.

Do improvements in system performance demonstrated by batch evaluations confer the same benefit for real users? We carried out experiments designed to investigate this question. After identifying a weighting scheme that gave maximum improvement over the baseline in a non-interactive evaluation, we used it with real users searching on an instance recall task. Our results showed the weighting scheme giving beneficial results in batch studies did not do so with real users. Further analysis did identify other factors predictive of instance recall, including number of documents saved by the user, document recall, and number of documents seen by the user.

Subfields: Computer Science Applications, Information Systems, Information Systems and Management, Information Retrieval & Search, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Science Applications

In Information Systems

In Information Systems and Management

In Information Retrieval & Search

In Evaluation & Benchmarks

Citations in the corpus

Information Retrieval Evaluation (2011, background): “[95] showed that better systems did lead to better interactive performance, however the differences were not statistically significant.”
Why batch and user evaluations do not give the same results (2001, result): “Our first experiments reported that not only did a system performing better in batch results not yield better results with a real user task, but that there was also no correlation between batch and searching results on most topics [7].”
What Makes a Good Answer? The Role of Context in Question Answering (2003, background): “Because system improvements, as measured by batch experiments , may not translate into actual benefits for end users (Hersh et al., 1999), the problem of computerhuman interaction should be studied in parallel.”
Toward a semantic granularity model for domain-specific information retrieval (2011, unknown): “1998], and TREC-8 [Beaulieu et al. 1999; Belkin et al. 1999; Fullter et al. 1999; Hersh and Over 1999; Hersh et al. 1999; Larson 1999; Yang et al. 1999].”
Assessor Differences and User Preferences in Tweet Timeline Generation (2015, background): “Early studies suggest that “better” systems (as measured by system-oriented metrics) don’t necessarily translate into better task performance [13, 29, 30].”
Methods for Evaluating Interactive Information Retrieval Systems with Users (2009, background): “There have been a series of studies that have compared results of batch-mode and user studies and found that systems which perform better in batch-mode studies do not always do so in user studies [129, 243, 275, 276].”
Test Collection Based Evaluation of Information Retrieval Systems (2010, background): “[122] were the first to try to correlate test collectionbased results with user behavior.”
A comparison of user and system query performance predictions (2010, background): “However, there has been related work examining and perhaps challenging the relationship between system-centered retrieval eﬀectiveness measures and actual user performance in traditional IR settings [1, 16, 28, 30].”
User performance versus precision measures for simple search tasks (2006, background): “Hersh et al. investigated whether batch and user evaluations give the same results for an instance recall task [13].”
Online Evaluation for Information Retrieval (2016, background): “A few studies have shown that there is not necessarily a strong correlation between offline evaluation metrics, such as mean average precision and precision at k, and online benefits for users [Hersh et al., 2000, Turpin and Hersh, 2001, Turpin and Scholer, 2006]. In particular, it is also demonstra”
Information Retrieval: A Biomedical and Health Perspective (2020, result): “One study compared various methods for presenting results to patients and found that relative and absolute risk reduction was more readily understood than number needed to treat (NNT), a common approach used to convey the magnitude of treatment benefit to clinicians [330].”