Bias and the limits of pooling for large collections

By Chris Buckley, Darrin L. Dimmick, Ian Soboroff, Ellen M. Voorhees. 2007

147 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Information Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

Citations in the corpus

On Building Fair and Reusable Test Collections using Bandit Techniques (2018, background): “Other approaches were needed when pooling also required too many judgments [1].”
TREC Deep Learning Track: Reusable Test Collections in the Large Data Regime (2021, background): “Leave-Out-Uniques (LOU) tests [5, 27] are a traditional way of analyzing the reusability of a collection.”
TREC 2017 Common Core Track Overview (2017, background): “This collection was the collection that ﬁrst demonstrated the limits of pooling [Buckley et al., 2007], in particular that a sufﬁcient pooling depth depends on collection size and so must grow as collection size grows.”
Overview of TREC 2021 (2021, unknown): “The test collection created in the Robust and HARD tracks in TREC 2005 showed that this point is not at some absolute pool size, but rather when pools are shallow relative to the number of documents in the collection [2].”
Overview of the TREC 2019 deep learning track (2020, background): “Leave-Out-Uniques (LOU) tests [Buckley et al., 2007, Zobel, 1998] are a way of analyzing the reusability of a collection.”
SogouT-16: A New Web Corpus to Embrace IR Research (2017, background): “The other two datasets, WT10g and Gov2 are somewhat dated and small given the rapid growth of the Web.”
Feeling lucky?: multi-armed bandits for ordering judgements in pooling-based evaluation (2016, background): “Pooling is fundamental in IR evaluation and, thus, has attracted much attention since its inception [19, 6, 5, 4, 10].”
Test Collection Based Evaluation of Information Retrieval Systems (2010, unknown): “[36].”
Q2D-Web: A Large-Scale Benchmark for Retrieval in Agentic RAG Systems (2026, unknown): “Subcorpus sampling inherits the bias induced by the retrieval systems used to construct the subcorpus: a collection can favor contributing retrievers and be biased against those that did not contribute (Buckley et al., 2007; Zobel, 1998).”
FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documents (2025, unknown): “Pooling [95, 5] is a predominant technique used in IR for selecting a subset of documents to be assessed for relevance, instead of assessing every document from the collection.”