Scaling Up Crowd-Sourcing to Very Large Datasets: A Case for Active Learning

By Barzan Mozafari, Purnamrita Sarkar, M. Franklin, Michael I. Jordan, Samuel Madden. 2014

201 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Science Applications, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sanjay Krishnan
2. Eugene Wu
3. Ihab F. Ilyas
4. Jiannan Wang
5. M. Franklin
6. Ken Goldberg
7. Xu Chu
8. Steven Euijong Whang

In Computer Science Applications

1. Sanjay Krishnan
2. Eugene Wu
3. Jiannan Wang
4. Xu Chu
5. M. Franklin
6. Ken Goldberg
7. Ihab F. Ilyas
8. Yuji Roh

In Evaluation & Benchmarks

1. Sanjay Krishnan
2. Eugene Wu
3. Jiannan Wang
4. Ken Goldberg
5. M. Franklin
6. Jian Pei
7. Alexander Toshev
8. Ihab F. Ilyas

Citations in the corpus

• AlphaClean: Automatic Generation of Data Cleaning Pipelines (2019, background): “This approach can be coupled with active learning [36] to learn an accurate model with the fewest possible number of examples.”
• A Survey on Data Collection for Machine Learning: A Big Data - AI Integration Perspective (2018, background): “[116] proposes active learning algorithms that can run in parallel.”
• BoostClean: Automated Error Detection and Repair for Machine Learning (2017, background): “This approach can be coupled with active learning [38] to learn an accurate model with the fewest possible number of examples.”
• The Unreasonable Effectiveness of Noisy Data for Fine-Grained Recognition (2015, background): “Research on using active learning in-the-wild on completely unlabeled data is rare; exceptions are [53, 35].”
• Data Cleaning: Overview and Emerging Challenges (2016, background): “In academia, there is a growing consensus that crowds are difficult to scale [35, 69, 62, 18, 82, 23, 37, 60, 61], and as a result several recent works employ Active Learning to prioritize queries to the crowd [38, 35, 59].”
• Crowdsourced Data Management: A Survey (2016, background): “E XISTING algorithms cannot effectively address computerhard tasks such as entity resolution [116], [118], [121], [122], sentiment analysis [74], [82], [139], and image recognition [104], [124], [129], which can benefit from the use of human cognitive ability.”
• Cleaning Crowdsourced Labels Using Oracles For Statistical Classification (2018, background): “There are three research topics in crowdsourcing related to our work: task assignment [4, 6, 13, 17, 26, 31, 46], truth inference [12,16,19,33,45], and active learning (from crowds) [18, 24, 28, 43].”
• ActiveClean: Interactive Data Cleaning While Learning Convex Loss Models (2016, background): “This approach can be coupled with active learning [27] to learn an accurate model with the fewest possible number of examples.”
• ActiveClean: Interactive Data Cleaning For Statistical Modeling (2016, background): “This approach can be coupled with active learning [25] to learn an accurate model with the fewest possible number of examples.”
• Knowledge Distillation Beyond Model Compression (2020, methodology): “Furthermore, to leverage the large amount of open-sourced data, automated label generation methods [23], [24] are employed which make use of user tags and keywords which inherently leads to noisy labels.”