On the Reliability of Test Collections for Evaluating Systems of Different Types
By Emine Yilmaz, Nick Craswell, Bhaskar Mitra, Daniel Fernando Campos. 2020
21 citations recorded by the source corpus.
As deep learning based models are increasingly being used for information retrieval, a major challenge is to ensure the availability of test collections for measuring their quality. Test collections are usually generated based on pooling results of various retrieval systems, but until recently this did not include deep learning systems. This raises a major challenge for reusable evaluation: Since deep learning based models use external resources (e.g. word embeddings) and advanced representations when compared to traditional methods, they may return different types of relevant document that were not identified in the original pooling. If so, test collections constructed using traditional methods could lead to biased and unfair evaluation results for deep learning systems. This paper uses simulated pooling to test the fairness and reusability of test collections, showing that especially when shallow pools (e.g. depth-10 pools) are used, pooling based on traditional systems only may lead to biased evaluation of deep learning systems.
Subfields: Artificial Intelligence, Information Systems
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sebastian Hofstätter
- Hamed Zamani
- Bhaskar Mitra
- Nick Craswell
- Jimmy J. Lin
- C. Clarke
- Allan Hanbury
- Xinyi Yan
In Information Systems
- Hamed Zamani
- Bhaskar Mitra
- Sebastian Hofstätter
- Nick Craswell
- C. Clarke
- Xinyi Yan
- Alexandra Vtyurina
- Negar Arabzadeh
Citations in the corpus
- Document Ranking with a Pretrained Sequence-to-Sequence Model (2020, neutral): “This finding confirms recent observations that test collections built before the advent of BERT-based rerankers place transformer-based models at a disadvantage (Yilmaz et al., 2020).”
- Conformer-Kernel with Query Term Independence for Document Retrieval (2020, methodology): “We anticipate that in the 2020 edition of the track, we will observe more submissions using neural methods for the full retrieval setting, which may further improve the reusability of the TREC benchmark [Yilmaz et al., 2020] for comparing these emerging family of approaches, and provide additional i”
- Shallow pooling for sparse labels (2021, methodology): “As new retrieval methods surface unjudged items with gain that would equals or exceed that of judged items, significant improvements can be missed [40].”