Evaluating sentence-level relevance feedback for high-recall information retrieval
By Haotian Zhang, Gordon V. Cormack, Maura R. Grossman, Mark D. Smucker. 2018
27 citations recorded by the source corpus.
This study uses a novel simulation framework to evaluate whether the time and effort necessary to achieve high recall using active learning is reduced by presenting the reviewer with isolated sentences, as opposed to full documents, for relevance feedback. Under the weak assumption that more time and effort is required to review an entire document than a single sentence, simulation results indicate that the use of isolated sentences for relevance feedback can yield comparable accuracy and higher efficiency, relative to the state-of-the-art baseline model implementation (BMI) of the AutoTAR continuous active learning (“CAL”) method employed in the TREC 2015 and 2016 Total Recall Track.
Subfields: Artificial Intelligence, Information Systems
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jimmy J. Lin
- Zeynep Akkalyoncu Yilmaz
- Wei Yang
- Haotian Zhang
- Shengjin Wang
- Rodrigo Nogueira
- Mark D. Smucker
- Andrew Yates
In Information Systems
- Jimmy J. Lin
- Zeynep Akkalyoncu Yilmaz
- Wei Yang
- Haotian Zhang
- Shengjin Wang
- Mark D. Smucker
- Maura R. Grossman
- Gordon V. Cormack
Citations in the corpus
- Simple Applications of BERT for Ad Hoc Document Retrieval (2019, methodology): “One rationale for this approach comes from Zhang et al. (2018b,a), who found that the “best” sentence or paragraph in a document provides a good proxy for document relevance.”
- Effective User Interaction for High-Recall Retrieval: Less is More (2018, background): “In simulation studies [39], we have found that even a single extracted sentence may be adequate for CAL to perform well.”
- A System for Efficient High-Recall Retrieval (2018, background): “Assessing the relevance of a document using a single paragraph can increase the total number of relevant documents found within a limited time and reduce review effort [10, 11].”
- Cross-Domain Modeling of Sentence-Level Evidence for Document Retrieval (2019, background): “Looking across all fine-tuning configurations, we see that the top-scoring sentence of each candidate document alone seems to be a good indicator of document relevance, corroborating the findings of Zhang et al. (2018a).”
- Applying BERT to Document Retrieval with Birch (2019, methodology): “The effectiveness of the system was further demonstrated in Zhang et al. (2018a).”