Effective User Interaction for High-Recall Retrieval: Less is More

By Haotian Zhang, Mustafa Abualsaud, Nimesh Ghelani, Mark D. Smucker, Gordon V. Cormack, Maura R. Grossman. 2018

29 citations recorded by the source corpus.

High-recall retrieval --- finding all or nearly all relevant documents --- is critical to applications such as electronic discovery, systematic review, and the construction of test collections for information retrieval tasks. The effectiveness of current methods for high-recall information retrieval is limited by their reliance on human input, either to generate queries, or to assess the relevance of documents. Past research has shown that humans can assess the relevance of documents faster and with little loss in accuracy by judging shorter document surrogates, e.g.\ extractive summaries, in place of full documents. To test the hypothesis that short document surrogates can reduce assessment time and effort for high-recall retrieval, we conducted a 50-person, controlled, user study. We designed a high-recall retrieval system using continuous active learning (CAL) that could display either full documents or short document excerpts for relevance assessment. In addition, we tested the value of integrating a search engine with CAL. In the experiment, we asked participants to try to find as many relevant documents as possible within one hour. We observed that our study participants were able to find significantly more relevant documents when they used the system with document excerpts as opposed to full documents. We also found that allowing participants to compose and execute their own search queries did not improve their ability to find relevant documents and, by some measures, impaired performance. These results suggest that for high-recall systems to maximize performance, system designers should think carefully about the amount and nature of user interaction incorporated into the system.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Evaluating sentence-level relevance feedback for high-recall information retrieval (2018, background): “In a recent study from Zhang et al (2018), they conducted a controlled 50-users study to evaluate using document excerpts (a single extracted paragraph from document) as relevance feedback in continuous active learning.”
Cross-Domain Sentence Modeling for Relevance Transfer with BERT (2019, background): “[76] which suggest that the most relevant sentence or paragraph in a document provides a good proxy for overall document relevance.”
Cross-Domain Modeling of Sentence-Level Evidence for Document Retrieval (2019, background): “Looking across all fine-tuning configurations, we see that the top-scoring sentence of each candi- date document alone seems to be a good indicator of document relevance, corroborating the findings of Zhang et al. (2018a).”
Applying BERT to Document Retrieval with Birch (2019, background): “The effectiveness of the system was further demonstrated in Zhang et al. (2018a).”