On Building Fair and Reusable Test Collections using Bandit Techniques

By Ellen M. Voorhees. 2018

34 citations recorded by the source corpus.

While test collections are a vital piece of the research infrastructure for information retrieval, constructing fair, reusable test collections for large data sets is challenging because of the number of human relevance assessments required. Various approaches for minimizing the number of judgments required have been proposed including a suite of methods based on multi-arm bandit optimization techniques. However, most of these approaches look to maximize the total number of relevant documents found, which is not necessarily fair, and they have only been demonstrated in simulation on existing test collections. The TREC 2017 Common Core track provided the opportunity to build a collection de novo using a bandit method. Doing so required addressing two problems not encountered in simulation: giving the human judges time to learn a topic and allocating the overall judgment budget across topics. The resulting modified bandit technique was used to build the 2017 Common Core test collection consisting of approximately 1.8 million news articles, 50 topics, and 30,030 judgments. Unfortunately, the constructed collection is of lower quality than anticipated: a large percentage of the known relevant documents were retrieved by only one team, and for 21 topics, more than a third of the judged documents are relevant. As such the collection is less reusable than desired. Further analysis demonstrates that the greedy approach common to most bandit methods can be unfair even to the runs participating in the collection-building process when the judgment budget is small relative to the (unknown) number of relevant documents.

Subfields: Artificial Intelligence, Computer Science Applications, Management Science and Operations Research, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Management Science and Operations Research

In Evaluation & Benchmarks

Citations in the corpus

Pretrained Transformers for Text Ranking: BERT and Beyond (2020, methodology): “ed by the amount of assessment resources available. This is referred to as top kpooling or pooling to depth k. Although NIST has also experimented with different approaches to constructing the pools [Voorhees, 2018], top kpooling remains the most popular. System results for each query are then prese”
Overview of the TREC 2019 deep learning track (2020, methodology): “Thus, in addition to the τ score between the rankings, we also report drops, the largest (negative) difference in ranks experienced by some run [Voorhees, 2018].”
On the Reliability of Test Collections for Evaluating Systems of Different Types (2020, methodology): “Various methods have been proposed in order to generate fair and reusable test collections with limited relevance labels [4, 5, 9, 10, 13].”
Towards Meaningful Statements in IR Evaluation: Mapping Evaluation Measures to Interval Scales (2021, background): “• Common Core track T26 [3]: 1,855,658 documents of the New York Times corpus; T26 provides 50 topics, each with multi-graded relevance judgments (not relevant, relevant, highly relevant); relevance judgements were done mixing depth-10 pools with multi-armed bandit approaches [57, 100]; 75 system ru”
TREC-COVID (2020, background): “Historically, when more than a third of the judged documents are relevant for a topic, it is highly likely that many more relevant documents that have not yet been identiﬁed remain in the collection [6].”