Combining human and machine intelligence in large-scale crowdsourcing

By Ece Kamar, Severin Hacker, Eric Horvitz. Adaptive Agents and Multi-Agent Systems, 2012

412 citations recorded by the source corpus.

We show how machine learning and inference can be harnessed to leverage the complementary strengths of humans and computational agents to solve crowdsourcing tasks. We construct a set of Bayesian predictive models from data and describe how the models operate within an overall crowd-sourcing architecture that combines the efforts of people and machine vision on the task of classifying celestial bodies defined within a citizens' science project named Galaxy Zoo. We show how learned probabilistic models can be used to fuse human and machine contributions and to predict the behaviors of workers. We employ multiple inferences in concert to guide decisions on hiring and routing workers to tasks so as to maximize the efficiency of large-scale crowdsourcing processes based on expected utility.

View Paper: https://www.semanticscholar.org/paper/57619d0962814f38a447fe9e26e8bcc74b5fc1bb

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Quizz: targeted crowdsourcing with a billion (potential) users (2014, methodology): “[21] also use a Markov Decision Process, in order to decide whether the answers provided by a user are promising enough to warrant a hiring decision.”
Best of both worlds: Human-machine collaboration for object annotation (2015, methodology): “Approaches such as [10, 46, 28, 62] use iterative improvement to perform a task with accuracy per unit of human cost.”
Towards Accountable AI: Hybrid Human-Machine Analyses for Characterizing System Failure (2018, background): “Most recently, there is increasing interest in human computation as a framework for enabling hybrid intelligence systems (Kamar, Hacker, and Horvitz 2012; Kamar 2016) as forms of fundamental human-machine collaboration.”
Learning from the Wisdom of Crowds by Minimax Entropy (2012, background): “1 Introduction There is an increasing interest in using crowdsourcing to collect labels for machine learning [19, 6, 21, 17, 20, 10, 13, 12].”
Disparate Interactions: An Algorithm-in-the-Loop Analysis of Fairness in Risk Assessments (2019, background): “Yet research within human-computer interaction and crowdsourcing suggests that alternative approaches could lead to a better synthesis of human and computer intelligence [8, 35, 39, 40].”
Crowdsourced Data Management: Industry and Academic Perspectives (2015, methodology): “[109] use POMDPs to study how to best utilize participation in voluntary crowdsourcing systems, specifically, Galaxy Zoo, an astronomical data set verified by workers.”
Mobile Crowd Sensing and Computing (2015, background): “For instance, Kamar et al. [2012] investigated how machine-learning techniques can be utilized to explore the complementary capabilities of humans and machineries to fulﬁll crowdsourcing tasks, for example, providing guidance on recruiting participants and assigning tasks based on certain metrics…”
From participatory sensing to Mobile Crowd Sensing (2014, background): “showed how learned probabilistic models can be used to fuse human and machine contributions and to predict the behaviors of workers in online crowdsourcing [17].”
The Principles and Limits of Algorithm-in-the-Loop Decision Making (2019, background): “Signi cant research in human-computer interaction has considered how to develop systems that e ectively integrate human and computer intelligence [40, 45].”
Revolt: Collaborative Crowdsourcing for Labeling Machine Learning Datasets (2017, background): “Prevention techniques include screening for crowdworkers capable of different tasks [18, 27], pre-training crowdworkers [19], maintaining quality while controlling for cost via dynamic task allocation [52, 8], or designing interfaces or payment structures to motivate good work [43, 47, 22].”
Updates in Human-AI Teams: Understanding and Addressing the Performance/Compatibility Tradeoff (2019, background): “While both human experts and machine-learned models are not perfect on a task like medical diagnosis or classifying objects in images, researchers have shown that their ideal combination could signiﬁcantly improve performance (Wang et al. 2016; Kamar, Hacker, and Horvitz 2012).”
Tuned Models of Peer Assessment in MOOCs (2013, background): “Explosion of interest in recent years has led to widespread applications of crowdsourcing [2, 11].”
Lean Multiclass Crowdsourcing (2018, unknown): “Alternatives to the DS model have also been proposed [28, 10, 37, 36, 26, 31, 11, 39, 2, 3].”
To Trust or to Think (2021, background): “, human+AI teams) were expected to perform better than either people or AIs alone [35, 36].”
Proxy tasks and subjective measures can be misleading in evaluating explainable AI systems (2020, background): “Because people and AI-powered systems have complementary strengths, many expected that human+AI teams would perform beer on decision-making tasks than either people or AIs alone [1, 21, 22].”
Scalable multi-label annotation (2014, background): “A growing body of work has studied how to estimate worker quality [9], how to combine results from multiple noisy annotators [20, 22, 19], how to model the trade-off between quality and cost [5], how to merge machine and human intelligence [10], as well as how to select the next best item to label [”
Artificial Intelligence and Collective Intelligence (2014, background): “Kamar et al. (Kamar, Kapoor, and Horvitz 2013) describe a similar approach applied to citizen science applications.”