A System for Efficient High-Recall Retrieval

By Mustafa Abualsaud, Nimesh Ghelani, Haotian Zhang, Mark D. Smucker, Gordon V. Cormack, Maura R. Grossman. 2018

47 citations recorded by the source corpus.

The goal of high-recall information retrieval (HRIR) is to find all or nearly all relevant documents for a search topic. In this paper, we present the design of our system that affords efficient high-recall retrieval. HRIR systems commonly rely on iterative relevance feedback. Our system uses a state-of-the-art implementation of continuous active learning (CAL), and is designed to allow other feedback systems to be attached with little work. Our system allows users to judge documents as fast as possible with no perceptible interface lag. We also support the integration of a search engine for users who would like to interactively search and judge documents. In addition to detailing the design of our system, we report on user feedback collected as part of a 50 participants user study. While we have found that users find the most relevant documents when we restrict user interaction, a majority of participants prefer having flexibility in user interaction. Our work has implications on how to build effective assessment systems and what features of the system are believed to be useful by users.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Systems, Information Retrieval & Search

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jimmy J. Lin
2. Zeynep Akkalyoncu Yilmaz
3. Wei Yang
4. Nick Craswell
5. Bhaskar Mitra
6. Emine Yilmaz
7. Daniel Fernando Campos
8. Sebastian Hofstätter

In Computer Vision and Pattern Recognition

1. Jimmy J. Lin
2. Zeynep Akkalyoncu Yilmaz
3. Wei Yang
4. Haotian Zhang
5. Nick Craswell
6. Bhaskar Mitra
7. Emine Yilmaz
8. Daniel Fernando Campos

In Information Systems

1. Jimmy J. Lin
2. Zeynep Akkalyoncu Yilmaz
3. Wei Yang
4. Emine Yilmaz
5. Bhaskar Mitra
6. Nick Craswell
7. Haotian Zhang
8. Ellen M. Voorhees

In Information Retrieval & Search

1. Jimmy J. Lin
2. Zeynep Akkalyoncu Yilmaz
3. Wei Yang
4. Rodrigo Nogueira
5. Emine Yilmaz
6. Nick Craswell
7. Bhaskar Mitra
8. Sebastian Hofstätter

Citations in the corpus

• Human Preferences as Dueling Bandits (2022, background): “Some TREC tracks, including the Deep Learning Track, use active learning methods [1] to expand the set of known relevant documents.”
• Effective User Interaction for High-Recall Retrieval: Less is More (2018, background): “The interface and functionality of search engine are also detailed in another paper [1].”
• Applying BERT to Document Retrieval with Birch (2019, methodology): “For the TREC Common Core Tracks in 2017 and 2018, a small group of researchers used HiCAL to find and judge relevant documents.”
• Evaluating sentence-level relevance feedback for high-recall information retrieval (2018, methodology): “Participants were asked to find as many relevant documents as possible within one hour using the HiCAL system (Abualsaud et al, 2018b; Zhang et al, 2017; Abualsaud et al, 2018a).”
• TREC Deep Learning Track: Reusable Test Collections in the Large Data Regime (2021, methodology): “We used the HiCAL system [2] to select the set of documents to be judged for each query once depth-10 pools were judged.”
• Overview of the TREC 2019 deep learning track (2020, methodology): “The assessor judges these pool documents first, then another 100 documents selected to be judged using the University of Waterloo’s HiCAL [Abualsaud et al., 2018] system.”
• Overview of TREC 2021 (2021, unknown): “The assessor judged these pool documents first, then another 20-25 documents selected to be judged using the University of Waterloo’s CAL [1] system.”