Feeling lucky?: multi-armed bandits for ordering judgements in pooling-based evaluation

76 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Towards Meaningful Statements in IR Evaluation: Mapping Evaluation Measures to Interval Scales (2021, background): “• Common Core track T26 [3]: 1,855,658 documents of the New York Times corpus; T26 provides 50 topics, each with multi-graded relevance judgments (not relevant, relevant, highly relevant); relevance judgements were done mixing depth-10 pools with multi-armed bandit approaches [57, 100]; 75 system ru”
TREC 2017 Common Core Track Overview (2017, methodology): “A particular bandit method, Losada et al. [2016]’s “MaxMean” or “MM” (stationary) method, was used in the Common Core track.”
On Building Fair and Reusable Test Collections using Bandit Techniques (2018, background): “Losada et al. [10] compared a variety of different bandit methods by forming traces of the documents encountered in simulations of the methods on existing test collections that were constructed through pooling (specifically, the TREC 5–8 ad hoc collections).”