Shepherding the crowd yields better work

By Steven W. Dow, Anand Pramod Kulkarni, Scott R. Klemmer, Bjoern Hartmann. Conference on Computer Supported Cooperative Work, 2012

370 citations recorded by the source corpus.

Subfields: Computer Science Applications, Management Science and Operations Research

View Paper: https://www.semanticscholar.org/paper/d4b451269c6e5bdc98de4bf9925f80500e4623a8

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Science Applications

In Management Science and Operations Research

Citations in the corpus

No Workflow Can Ever Be Enough (2017, background): “For example, research has shown that providing feedback to crowd workers can lead to better results [20] and that balancing personalities of crowd workers can lead to more effective crowd teams with higher quality outputs and higher quality communication [45].”
On the State of Reporting in Crowdsourcing Experiments and a Checklist to Aid Current Practices (2021, background): “Other strategies involve imposing quality by design, such as adding feedback loops (where workers self-assess or receive external feedback) [22], or making workers collaborate [10, 16, 67].”
COCO Attributes: Attributes for People, Animals, and Objects (2016, methodology): “3 we employ proven techniques, such as text-mining, image comparison tasks, and crowd shepherding, to find the 196 attributes we later use to label the dataset [5,6,7,8,9].”
Revolt: Collaborative Crowdsourcing for Labeling Machine Learning Datasets (2017, background): “…crowdsourcing has been employed for a variety of purposes including minimizing reaction time in crowd-powered user interfaces [7, 6, 9, 34, 36], increasing the speed of data collection [35, 31], and improving data quality by incorporating expert interventions to guide novice workers [11, 20, 28].”
Chain Reactions: The Impact of Order on Microtask Chains (2016, background): “While iterating on similar tasks can help people build familiarity and expertise [15][34], long chains of similar tasks can lead to boredom and fatigue [29][38].”
What Ingredients Make for an Effective Crowdsourcing Protocol for Difficult NLU Data Collection Tasks? (2021, methodology): “There are some small-scale studies that use writing tasks, like writing product reviews, to compare crowdsourcing methodologies (Dow et al., 2012).”
Beyond Mechanical Turk: An Analysis of Paid Crowd Work Platforms (2015, background): “Many studies discuss quality concerns and possible solutions (Dow, Kulkarni, Klemmer, & Hartmann, 2012).”
A Glimpse Far into the Future: Understanding Long-term Crowd Worker Accuracy (2016, background): “Further work should be conducted to not only retain a worker pool, but also examine methods of identifying good workers [21] and more direct interventions for training poorly performing workers [13, 22].”
Bootstrapping Fine-Grained Classifiers : Active Learning with a Crowd in the Loop (2013, result): “Similar research into parallelizing crowd micro-tasks has recently been described by [13] [4] [7].”
A Glimpse Far into the Future: Understanding Long-term Crowd Worker Quality (2016, background): “Further work should be conducted to not only retain a worker pool, but also examine methods of identifying good workers [22] and more direct interventions for training poorly performing workers [13, 23].”
Crowdsourced Data Management: Industry and Academic Perspectives (2015, background): “There is also work from the HCI community on improved interfaces for task requesters to supervise and evaluate workers, including visualizing worker behavior [164], supervising their work [77], performing analytics on worker retention and fatigue [95], and relinquishing control of GUIs to remote wor”
Peer and self assessment in massive online classes (2013, background): “Prior work has demonstrated that feedback improves the quality of crowd work [Dow et al. 2012], but can it help raters overcome their (possibly unintentional) grading bias?”