Crowdsourcing graphical perception: using mechanical turk to assess visualization design

By Jeffrey Heer, Michael Bostock. International Conference on Human Factors in Computing Systems, 2010

919 citations recorded by the source corpus.

Subfields: Computer Science Applications, Computer Vision and Pattern Recognition, Experimental and Cognitive Psychology

View Paper: https://www.semanticscholar.org/paper/55d3281f6b34c50df975b7261044689bf73ec610

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Science Applications

In Computer Vision and Pattern Recognition

In Experimental and Cognitive Psychology

Citations in the corpus

Beyond Mechanical Turk: An Analysis of Paid Crowd Work Platforms (2015, background): “How are alternative organizational structures determined and implemented across varying task types and complexity (Noronha et al., 2011; Heer & Bostock, 2010; Lasecki et al., 2012)?”
TurKit: human computation algorithms on mechanical turk (2010, background): “Individual systems have also been studied and explored in academic literature, including Games with a Purpose [1], Wikipedia [9] [15], and Mechanical Turk [7] [8] [12] [13] [14].”
Schelling points on 3D surface meshes (2012, background): “, [Heer and Bostock 2010]), it is valuable to ask whether the data we gathered from the AMT is representative of that which would be collected from a controlled group of participants in a laboratory environment.”
Comparing the Similarity of Responses Received from Studies in Amazon’s Mechanical Turk to Studies Conducted Online and with Direct Recruitment (2015, methodology): “The average time to complete a task is higher in AMT studies compared to laboratory setting which indicates a possible problem of controlling the participants’ environment [5], although there are no set rules on how long a researcher should give subjects to complete tasks [30].”
Crowdsourced enumeration queries (2013, neutral): “samplers” effect [18].”
Turkers, Scholars, "Arafat" and "Peace": Cultural Communities and Algorithmic Gold Standards (2015, result): “Similar results have been seen in studies in psychology [7], translation [5], graphical perception [21], and a number of other areas.”
A Survey of Crowdsourcing Systems (2011, background): “A taxonomy of crowdsourcing work were proposed to solve the problem of geometric reasoning on MTurk [39], [28].”
Mini-crowdsourcing end-user assessment of intelligent assistants: A cost-benefit study (2011, background): “A similar study of Mechanical Turk’s viability for assessing visualization design showed that crowdsourcing could contribute new insights for such designs [7].”
Empirical Studies in Information Visualization: Seven Scenarios (2012, positive): “Interested readers are directed to validation studies of the method [30, 45].”
A Principled Way of Assessing Visualization Literacy (2014, neutral): “While the validity of using this platform may be debated, due to lack of control over particular experimental conditions [19], we considered it best to perform our calibration using the results of a wide variety of people.”
Inside the Turk (2014, background): “…attentiveness ex ante by including tasks that signal to workers that their subsequent responses will be scrutinized (e.g., tasks in which participants must answer factually verifiable questions; Heer & Bostock, 2010; Kittur, Chi, & Suh, 2008) or require commitment (Rand, Greene, & Nowak, 2012).”