Shallow pooling for sparse labels

By Negar Arabzadeh, Alexandra Vtyurina, Xinyi Yan, C. Clarke. 2021

56 citations recorded by the source corpus.

Recent years have seen enormous gains in core information retrieval tasks, including document and passage ranking. Datasets and leaderboards, and in particular the MS MARCO datasets, illustrate the dramatic improvements achieved by modern neural rankers. When compared with traditional information retrieval test collections, such as those developed by TREC, the MS MARCO datasets employ substantially more queries—thousands vs. dozens – with substantially fewer known relevant items per query—often just one. For example, 94% of the nearly seven thousand queries in the MS MARCO passage ranking development set have only a single known relevant passage, and no query has more than four. Given the sparsity of these relevance labels, the MS MARCO leaderboards track improvements with mean reciprocal rank (MRR). In essence, the known relevant item is treated as the “right answer” or “best answer”, with rankers scored on their ability to place this item as high in the ranking as possible. In working with these sparse labels, we have observed that the top items returned by a ranker often appear superior to judged relevant items. Others have reported the same observation. To test this observation, we employed crowdsourced workers to make preference judgments between the top item returned by a modern neural ranking stack and a judged relevant item for the nearly seven thousand queries in the passage ranking development set. The results support our observation. If we imagine a hypothetical perfect ranker under MRR, with a score of 1 on all queries, our preference judgments indicate that a searcher would prefer the top result from a modern neural ranking stack more frequently than the top result from the hypothetical perfect ranker, making our neural ranker “better than perfect”. To understand the implications for the leaderboard, we pooled the top document from available runs near the top of the passage ranking leaderboard for over 500 queries. We employed crowdsourced workers to make preference judgments over these pools and re-evaluated the runs. Our results support our concerns that current MS MARCO datasets may no longer be able to recognize genuine improvements in rankers. In future, if rankers are measured against a single answer, this answer should be the best answer or most preferred answer, and maintained with ongoing judgments. Since only the best known answer is required, this ongoing maintenance might be performed with shallow pooling. When a previously unjudged document is surfaced as the top item in a ranking, it can be directly compared with the previous best known answer.

Subfields: Artificial Intelligence, Information Systems, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Model Architectures

Citations in the corpus

Hindsight: Posterior-guided training of retrievers for improved open-ended generation (2021, positive): “…when compared with Wizard of Wikipedia, a fact we largely attribute— based on manual inspection—to the presence of many false negatives, i.e., passages that contain the answer but aren’t marked as gold passages, which is consistent with the ﬁndings from related studies Arabzadeh et al. (2021).”
Human Preferences as Dueling Bandits (2022, methodology): “pairs in a judging pool will be judged once [2, 7, 32].”
Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agent (2023, methodology): “Our findings can be summarized as follows: (i) Permutation distillation outperforms the supervised counterpart on both TREC and BEIR datasets, potentially because ChatGPT’s relevance judgments are more comprehensive than MS MARCO labels (Arabzadeh et al., 2021).”
Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, background): “This issue becomes more severe for hard negatives, which are more likely to be false negatives [74], [214], [215] (i.”
Introducing Neural Bag of Whole-Words with ColBERTer: Contextualized Late Interactions using Enhanced Reduction (2022, background): “Our main aim is to present an analysis grounded in robust evaluation [50, 58] that does not fall for common problematic shortcuts in IR evaluation like influence of effect sizes [11, 52], relying on too shallow pooled collections [2, 31, 53], not accounting for pool bias in old collections [5, 40, 4”
Q2D-Web: A Large-Scale Benchmark for Retrieval in Agentic RAG Systems (2026, unknown): “…false negatives are common under sparse labels: on MS MARCO, 70% of manually inspected top-retrieved but unlabeled passages were in fact relevant (Qu et al., 2021), and preference judgments rated the top results of a modern ranker above the officially judged relevant items (Arabzadeh et al., 2022).”