Pairwise ranking aggregation in a crowdsourced setting

By Xi Chen, Paul N. Bennett, Kevyn Collins-Thompson, Eric Horvitz. Web Search and Data Mining, 2013

373 citations recorded by the source corpus.

Subfields: Computer Science Applications, Management Science and Operations Research

View Paper: https://www.semanticscholar.org/paper/199dcbb1e5287eedb458c867b171cc83c06b0d2a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Science Applications

In Management Science and Operations Research

Citations in the corpus

The Sample Complexity of Best-k Items Selection from Pairwise Comparisons (2020, background): “hen the comparisons are generated, the ranking problems can be divided into two classes: passive ranking (e.g., (Park et al. ,2015;Shah et al. 2017)) and active ranking (e.g., (Pfeiffer et al. ,2012 ;Chen et al. 2013 Falahatgar et al. ,2017a;Ren et al. 2019)). In passive ranking, the learner ﬁrst ha”
Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons (2023, background): “On the other hand, in the literature of ranking, most of the theoretical work focuses on the tabular case where the rewards for different actions are uncorrelated (Feige et al., 1994; Shah et al., 2015; Shah and Wainwright, 2017; Heckel et al., 2018; Mao et al., 2018; Jang et al., 2017; Chen et al.,”
Spectral Method and Regularized MLE Are Both Optimal for Top-$K$ Ranking (2017, neutral): “…sampling (Braverman and Mossel, 2008; Jamieson and Nowak, 2011; Ailon, 2012; Busa-Fekete et al., 2013; Mohajer and Suh, 2016; Heckel et al., 2016; Agarwal et al., 2017), for the crowdsourcing scenario (Chen et al., 2013; Osting et al., 2016), and in the adversarial setting (Suh et al., 2017).”
Neural Sentence Ordering (2016, background): “The score of sentences in order o can be calculated as a log-likelihood maximization problem (Chen et al., 2013):”
Preference-based Online Learning with Dueling Bandits: A Survey (2018, background): “The task is to approximate an underlying target ranking on the basis of these pairwise comparisons, which are possibly noisy and partially noncoherent (Chen et al., 2013).”
Preferences on a Budget: Prioritizing Document Pairs when Crowdsourcing Relevance Judgments (2022, background): “…et al. [5] that brought up again the opportunity, in recent years there has been work looking at different open challenges around the use of preference judgments including how to reliably perform IR evaluation under the assumption that preference judgments are available [2, 4, 6, 7, 23, 34, 35].”
Estimation from Pairwise Comparisons: Sharp Minimax Bounds with Topology Dependence (2015, background): “Chen et al. (2013) consider a crowdsourcing setup where the outcome depends on the worker’s expertise.”
A Survey of Preference-Based Online Learning with Bandit Algorithms (2014, background): “The task is to approximate an underlying target ranking on the basis of these pairwise comparisons, which are possibly noisy and partially inconsistent [17].”
Crowdsourced Data Management: A Survey (2016, methodology): “[23] propose the CrowdBT model, assuming that each worker has a quality hw as discussed in Worker Probability (Section 4).”
Interestingness Prediction by Robust Learning to Rank (2014, background): “(2) Outliers – it is well known that crowdsourced data are noisy [6,43,30].”
Juxtapeer: Comparative Peer Review Yields Higher Quality Feedback and Promotes Deeper Reflection (2018, methodology): “Optimal submission pairs for review with Crowd-BT To pick similar submissions for comparison, Juxtapeer uses a modified version of the Crowd-BT [8] active learning algorithm.”
Pairwise Crowd Judgments: Preference, Absolute, and Ratio (2018, background): “Compared to ordinal relevance scales, pairwise preferences help to reduce the complexity and increase the consistency of relevance assessing [7], reducing the cost of collecting valid relevance judgments with a smaller number of tied relevance scores.”