Human computation: a survey and taxonomy of a growing field

By Alexander J. Quinn, B. Bederson. 2011

939 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Science Applications, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Human-Computer Interaction

Citations in the corpus

CrowdWorkSheets: Accounting for Individual and Collective Identities Underlying Crowdsourced Dataset Annotation (2022, background): “Human computation refers to the practice of tapping into human intelligence and cognition as computational elements within an information processing system design, often done on a large global scale [41].”
The Language Demographics of Amazon Mechanical Turk (2014, background): “With an increasing number of active crowd workers located outside of the United States, there is even the potential to reach fluent speakers of lower resource languages.”
An innovative web-based collaborative platform for video annotation (2013, methodology): “…concern of crowdsourcing solutions is the quality control over annotators, which has been tackled with different strategies that can be summarized [31] as: Task Redundancy (ask multiple users to annotate the same data), User Reputation and Groundtruth seeding (i.e. coupling ground truth with test…”
Crowdsourced Data Management: Industry and Academic Perspectives (2015, background): “There are many conflicting opinions [153] on how to define crowdsourcing, and whether crowdsourcing is indeed the same concept as human computation.”
CrowdScreen: algorithms for filtering data with humans (2012, background): “Crowdsourcing enables programmers to write procedures that employ Human Computation [1, 7, 18] at a large scale in order to solve problems that are hard to solve algorithmically, such as understanding text, sound, video and images.”
Ethical norms and issues in crowdsourcing practices: A Habermasian analysis (2018, background): “Crowdsourcing is sometimes conceptualized as a form of “human computation” (Law and von Ahn 2011; Quinn and Bederson 2011) using people (i.e., “men in the machine”) as part of algorithms or HCI systems for problems that cannot be efficiently solved by computer.”
The Psychological Well-Being of Content Moderators: The Emotional Labor of Commercial Moderation and Avenues for Improving Support (2021, background): “While AI will continue to improve, the use of human computation enables companies to deliver greater system capabilities today [8, 19, 90, 121], what Gray and Suri call AI’s “last mile” [64].”
Mobile Crowd Sensing and Computing (2015, result): “Similar to other crowdsourcing systems [Quinn and Bederson 2011; Yuen et al. 2011], broadly two types of incentives can be used in MCSC applications: intrinsic incentives or ﬁnancial incentives.”
A Survey of Crowdsourcing Systems (2011, background): “In some cases, workers are not motivated by rewards, but they work for fun or altruis [66].”
Turkomatic: automatic recursive task and workflow design for mechanical turk (2011, methodology): “The TurKit project provides tools for deploying arbitrary iterative tasks on Mechanical Turk to enhance quality and eliminate redundant computation (Little 2009, 2010).”
No Workflow Can Ever Be Enough (2017, background): “While tasks are typically given to workers, some such as image labeling [49] and translation [29] may be automatically completed by computers or a combination of humans and machines [51, 66].”
Soylent: a word processor with a crowd inside (2010, background): “Our work also contributes the Find-Fix-Verify pattern to improve the quality of such proofreading services.”
Evaluation in artificial intelligence: from task-oriented to ability-oriented measurement (2017, background): “…Beer, 2010), machine-enhanced humans (Cohen, 2013), human-enhanced machines (von Ahn, 2009), other kind of hybrids and, most especially, collectives (Quinn and Bederson, 2011) of any of the former, suggest that the distinction between animals, humans and machines is not only inappropriate, but no…”
Crowdsourcing with Smartphones (2012, background): “In [13], the incen tiv s used for the crowd are also studied, which can be one or more from: pay, altruism, enjoyment, reputation , among others.”
Beyond Mechanical Turk: An Analysis of Paid Crowd Work Platforms (2015, background): “Quinn and Bederson (2011) review human computation with respect to motivation, quality control, aggregation, human skill, process order, and task-request cardinality.”