Crowdsourcing for relevance evaluation

By Omar Alonso, Daniel E. Rose, Benjamin Stewart. SIGF, 2008

404 citations recorded by the source corpus.

Subfields: Communication, Information Systems

View Paper: https://www.semanticscholar.org/paper/ac9fb9c2878fa2882996be5a4b4859a70600c91d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Communication

In Information Systems

Citations in the corpus

Test Collection Based Evaluation of Information Retrieval Systems (2010, methodology): “[7] described their use of the Amazon Mechanical Turk system to obtain relevance judgments.”
Methods for Evaluating Interactive Information Retrieval Systems with Users (2009, background): “Recently, many researchers have relied on crowdsourcing and mechanical Turk to recruit people to make relevance assessments via the Web [7].”
How Much Spam Can You Take? An Analysis of Crowdsourcing Results to Increase Accuracy (2011, background): “Relevance judgments can be crowdsourced on the Internet by using anonymous web users (known as workers) as non-expert annotators [1].”
A Survey of Crowdsourcing Systems (2011, background): “Alonso et al. [6] proposed crowdsourcing for relevance evaluation, so that each crowdsourcing work perform a small evaluation task.”
CDAS: A Crowdsourcing Data Analytics System (2012, background): “Typical tasks include image annotation [21][18], information retrieval [1][8] and natural language processing [3][12][17].”
A survey of the use of crowdsourcing in software engineering (2017, methodology): “…(Brabham et al., 2009; Misra et al., 2014), weather forecasting (Chatfield and Brajawidagda, 2014; Muller et al., 2015), information retrieval (Alonso et al., 2008; Lease and Yilmaz, 2012), and software engineering (Stol and Fitzgerald, 2014c; Breaux and Schaub, 220 2014; Schiller and Ernst,…”
Evaluation on crowdsourcing research: Current status and future direction (2012, background): “Some researchers from other disciplines also applied the crowdsourcing to support, mediate, and facilitate their work on user studies (Alonso et al. 2008; Eckert et al. 2010; Franklin et al. 2011; Kittur et al. 2008; Marcus et al. 2011), cataloging (Holley 2009), and transportation plan (Brabham…”
Pairwise Crowd Judgments: Preference, Absolute, and Ratio (2018, background): “Workers are paid modest amounts of money to complete human intelligence tasks (HITs) published by task creators [1], and it is natural to consider making use of these services for relevance judgments [11].”
Crowdsourced Data Management: Industry and Academic Perspectives (2015, background): “Most companies with a search engine, e.g., Bing [11], Google [9], and Yahoo!, include crowd workers in evaluating the performance of their search algorithms [26].”
Who are the Turkers? Worker Demographics in Amazon Mechanical Turk (2009, methodology): “MTurk has becoming increasingly popular as a tool for research, being used for performing user studies [4,8], image labeling [11], natural language processing [10], and relevance evaluation [1].”
GeoCrowd: enabling query answering with spatial crowdsourcing (2012, background): “Examples of such applications are image search a) b) Figure 7: Eﬀect of R - Synthetic data [38], natural language annotations [31], video and image annotations [14, 32], social games [20, 35], graph search [28], and search relevance [11].”
Who are the crowdworkers?: shifting demographics in mechanical turk (2010, methodology): “MTurk is increasingly popular as a tool for research, being used for performing user studies [8,13], image labeling [7,18], natural language processing [17], and relevance evaluation [4], as well as replicating classical economic and social science experiments [14].”
CrowdScreen: algorithms for filtering data with humans (2012, methodology): “Relevance judgments for search results [6], where the task is to determine whether a web-page is relevant or not for a given websearch query, is another instance.”