Crowdsourcing user studies with Mechanical Turk

By Aniket Kittur, Ed H. Chi, Bongwon Suh. 2008

2,165 citations recorded by the source corpus.

Subfields: Communication, Computer Science Applications

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Communication

In Computer Science Applications

Citations in the corpus

Explainable Recommendation: A Survey and New Perspectives (2018, background): “derived from the responses of the subjects (Kittur et al., 2008).”
In AI We Trust? Factors That Influence Trustworthiness of AI-infused Decision-Making Processes (2019, methodology): “de this facet from our analysis. We also included a manipulation check in the form of attention questions to ensure the quality of survey responses on Mechanical Turk, as recommended by Kittur et al. [22]. Each participant was asked two questions about the specific details of each scenario (4 questi”
Cheap and Fast – But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks (2008, neutral): “Kittur et al. (2008) compared AMT evaluations of Wikipedia article quality against experts, finding validation tests were important to ensure good results.”
Corpus Annotation through Crowdsourcing: Towards Best Practice Guidelines (2014, methodology): “Both Laws et al. (2011) and Kittur et al. (2008) have extended their interfaces with explicitly veriﬁable questions which force the users to process the content and also signal to the workers that their answers are being scrutinized.”
HYPE: A Benchmark for Human eYe Perceptual Evaluation of Generative Models (2019, neutral): “Human evaluations, such as those given on Amazon Mechanical Turk [49, 14], remain ad-hoc because “results change drastically” [52] based on details of the task design [36, 34, 27].”
Who are the Turkers? Worker Demographics in Amazon Mechanical Turk (2009, methodology): “Workers (or "Turkers") who log into the MTurk website are able to pick and choose which tasks they perform (after previewing the HIT), thus creating a microtask marketplace [8].”
Effect of trapping questions on the reliability of speech quality judgments in a crowdsourcing paradigm (2015, background): “With Trapping T2 and Trapping T3 we examined the assumption that the likelihood of cheating decreases if the effort to conceal cheating is as high as the effort to accurately complete a task [11].”
Who are the crowdworkers?: shifting demographics in mechanical turk (2010, methodology): “Workers (or "Turkers") who log into the MTurk website are able to choose which tasks they perform (after previewing the HIT), creating a micro-task marketplace [13].”
Mini-crowdsourcing end-user assessment of intelligent assistants: A cost-benefit study (2011, background): “A study of Mechanical Turk’s suitability for usability assessment found that crowds of end users have great potential for rapidly collecting user measurements at a low cost [8].”
CROWDMOS: An approach for crowdsourcing mean opinion score studies (2011, neutral): “Index Terms— crowdsourcing, subjective quality, mean opinion score, MOS, MUSHRA, mechanical turk, crowdMOS.”
Evaluating Recommender Systems with User Experiments (2015, neutral): “workers with a high reputation [58, 92].”
Serf and turf: crowdturfing for fun and profit (2011, background): “This includes studies of worker demographics [1 4, 28], task pricing [7, 13], and even meta-studies on how to use Mechanical Turk to conduct user studies [18].”
Using Mechanical Turk to Study Clinical Populations (2013, positive): “In particular, on MTurk, including questions with factually verifiable answers improves the reliability of subjective judgments (Kittur, Chi, & Suh, 2008).”
Shepherding the crowd yields better work (2012, background): “Timeliness: When should feedback be shown? In micro-task labor, many workers perform tasks for a short time (seconds to minutes), and then move to other tasks by other employers [12,19].”
Examining Crowd Work and Gig Work Through the Historical Lens of Piecework (2017, background): “For years, such labor was limited to information work such as data annotation and surveys [82, 161, 166, 51, 119].”
"Cargo Cult" science in traditional organization and information systems survey research: A case for using nontraditional methods of data collection, including Mechanical Turk and online panels (2016, background): “(Note that, of these four papers, one was a conceptual piece, whereas the remaining three used a single MTurk provides quicker data collection around the clock regardless of researchers’ and participants’ locations (Buhrmester et al., 2011; Goodman et al., 2013; Kittur et al., 2008).”
Comparing the Similarity of Responses Received from Studies in Amazon’s Mechanical Turk to Studies Conducted Online and with Direct Recruitment (2015, background): “Kittur and colleagues [33] compared quantitative user ratings and qualitative feedback regarding the quality of Wikipedia articles given by highly experienced Wikipedia users to answers gathered through AMT.”
Training deep networks for facial expression recognition with crowd-sourced label distribution (2016, background): “, data or dummy questions with known answers [3, 9, 15].”