Revolt: Collaborative Crowdsourcing for Labeling Machine Learning Datasets

By Joseph Chee Chang, Saleema Amershi, Ece Kamar. 2017

273 citations recorded by the source corpus.

Crowdsourcing provides a scalable and efficient way to construct labeled datasets for training machine learning systems. However, creating comprehensive label guidelines for crowdworkers is often prohibitive even for seemingly simple concepts. Incomplete or ambiguous label guidelines can then result in differing interpretations of concepts and inconsistent labels. Existing approaches for improving label quality, such as worker screening or detection of poor work, are ineffective for this problem and can lead to rejection of honest work and a missed opportunity to capture rich interpretations about data. We introduce Revolt, a collaborative approach that brings ideas from expert annotation workflows to crowd-based labeling. Revolt eliminates the burden of creating detailed label guidelines by harnessing crowd disagreements to identify ambiguous concepts and create rich structures (groups of semantically related items) for post-hoc label decisions. Experiments comparing Revolt to traditional crowdsourced labeling show that Revolt produces high quality labels without requiring label guidelines in turn for an increase in monetary cost. This up front cost, however, is mitigated by Revolt's ability to produce reusable structures that can accommodate a variety of label boundaries without requiring new data to be collected. Further comparisons of Revolt's collaborative and non-collaborative variants show that collaboration reaches higher label accuracy with lower monetary cost.

Subfields: Artificial Intelligence, Computer Science Applications

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

Citations in the corpus

A Survey on Data Collection for Machine Learning: A Big Data - AI Integration Perspective (2018, background): “Revolt [115] is a system that attempts to fix this problem through collab-”
Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need? (2018, neutral): “, [22, 24, 39, 66]).”
Garbage in, garbage out?: do machine learning application papers in social computing report where human-labeled training data comes from? (2019, neutral): “There have also been many new software platforms developed to support more micro-level annotation or labeling at scale, including in citizen science, linguistics, content moderation, and more general-purpose use cases [5, 9, 21, 34, 41, 47].”
Errudite: Scalable, Reproducible, and Testable Error Analysis (2019, background): “Furthermore, due to subjectivity, the labels themselves are not precisely deﬁned (Chang et al., 2017).”
Crowdsourcing Subjective Tasks: The Case Study of Understanding Toxicity in Online Discussions (2019, methodology): “[8] explores limiting guidelines in favor of post-hoc label decisions, which reduces training time and surfaces potentially unanticipated ambiguity in”
“Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI (2021, unknown): “in making use of crowdsourcing to generate ML data [25, 128], to support creation of better task designs for raters [59], compute interrater reliability, design incentives [50], and improve the quality of crowdsourced data [30], though these areas are less well known in the ML community [122].”
Artsheets for Art Datasets (2021, background): “[16] Joseph Chee Chang, Saleema Amershi, and Ece Kamar.”