Methods for Evaluating Interactive Information Retrieval Systems with Users

By Diane Kelly. 2009

709 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Information Systems, Information Systems and Management, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jimmy J. Lin
2. Andrew Yates
3. Rodrigo Nogueira
4. Katja Hofmann
5. Filip Radlinski
6. Zeynep Akkalyoncu Yilmaz
7. Sebastian Hofstätter
8. Lihong Li

In Information Systems

1. Rodrigo Nogueira
2. Andrew Yates
3. Jimmy J. Lin
4. Katja Hofmann
5. Lihong Li
6. Filip Radlinski
7. Donna K. Harman
8. Lucie Flekova

In Information Systems and Management

1. Rodrigo Nogueira
2. Andrew Yates
3. Donna K. Harman
4. Katja Hofmann
5. Jimmy J. Lin
6. Filip Radlinski
7. Lucie Flekova
8. Lihong Li

In Evaluation & Benchmarks

1. Rodrigo Nogueira
2. Andrew Yates
3. Jimmy J. Lin
4. Katja Hofmann
5. Donna K. Harman
6. Filip Radlinski
7. Lucie Flekova
8. Lihong Li

Citations in the corpus

• Pretrained Transformers for Text Ranking: BERT and Beyond (2020, methodology): “less, there are other evaluation paradigms worth noting: interactive evaluations place humans “in the loop” and are necessary to understand the important role of user behavior in information seeking [Kelly, 2009]. Online services with substantial numbers of users can engage in experimentation using ”
• Information Retrieval Evaluation (2011, methodology): “It should be noted that the scope of this chapter is narrow; the recent tutorial (Methods for Evaluating Interactive Information Systems with Users) by Diane Kelly [111] provides detailed methodologies for organizing user studies, data collection and analysis techniques, etc.”
• Returning the N to NLP: Towards Contextually Personalized Classification Models (2020, methodology): “We can take inspiration from subjective measures used in evaluating spoken dialogue systems, such as A/B testing (Kohavi et al., 2014), customer satisfaction (Kelly et al., 2009; Kiseleva et al., 2016) or interestingness (Harrison et al., 2019; Oraby et al., 2018).”
• Online Evaluation for Information Retrieval (2016, methodology): “More details of these alternatives were discussed by Kelly et al. [2014]. The standard approach to online evaluation is server-side logging, implementing a full search system: When a user arrives at the search”
• From Semantic Retrieval to Conversational Agent: A Web-Based RAG Architecture for Interactive System Dynamics Modeling (2026, unknown): “We evaluated the system by simulating specific levels of user expertise and search strategies, following established methodologies for interactive information retrieval evaluation [21].”