TurKit: human computation algorithms on mechanical turk

By Greg Little, Lydia B. Chilton, Max Goldman, Rob Miller. 2010

384 citations recorded by the source corpus.

Mechanical Turk (MTurk) provides an on-demand source of human computation. This provides a tremendous opportunity to explore algorithms which incorporate human computation as a function call. However, various systems challenges make this difficult in practice, and most uses of MTurk post large numbers of independent tasks. TurKit is a toolkit for prototyping and exploring algorithmic human computation, while maintaining a straight-forward imperative programming style. We present the crash-and-rerun programming model that makes TurKit possible, along with a variety of applications for human computation algorithms. We also present case studies of TurKit used for real experiments across different fields.

Subfields: Artificial Intelligence, Computer Science Applications, Information Systems, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Information Systems

In Human-Computer Interaction

Citations in the corpus

ToTTo: A Controlled Table-To-Text Generation Dataset (2020, methodology): “…of editing exemplar sentences has been used in semiparametric generation models (Guu et al., 2018; Pandey et al., 2018; Peng et al., 2019) and crowd-sourcing small, iterative changes to text has been shown to lead to higher-quality data and a more robust annotation process (Little et al., 2010).”
A Survey on Data Collection for Machine Learning: A Big Data - AI Integration Perspective (2018, methodology): “TurKit [45] allows users to write scripts that”
Mechanical Novel: Crowdsourcing Complex Work through Reflection and Revision (2016, methodology): “These microtask systems present work in an assembly line-like structure called a workﬂow [6], using mechanisms such as iteration [25], clustering [9], voting [26], and other patterns for splitting work [5, 19, 21].”
Examining Crowd Work and Gig Work Through the Historical Lens of Piecework (2017, background): “In contrast, today system– designers can share, modify, and instantiate environments like sites of labor in a few lines of code [95, 98].”
Sequential line search for efficient visual design optimization by crowds (2017, background): “Little et al. [2010] described the concept of human computation algorithms , in which crowdsourced human processors are incorporated into algorithms in the form of function calls.”
Answering visual questions with conversational crowd assistants (2013, background): “Prior systems (such as [13, 10, 1]) focused on discrete micro-tasks that workers were asked to complete, whereas in Legion, workers are asked to participate in a closed-loop interaction with the system they are controlling.”
Turkomatic: automatic recursive task and workflow design for mechanical turk (2011, background): “Bigham et al.'s VizWiz is capable of handling openended , natural-language requests from its users, but doesn't attempt to parallelize these queries or handle tasks more complex than short requests (Bigham 2010).”
Cascade: crowdsourcing taxonomy creation (2013, methodology): “The idea of using small units of human labor in an algorithm was introduced by TurKit [15] and demonstrated on simple workflows such as iteratively improving image descriptions.”
No Workflow Can Ever Be Enough (2017, methodology): “Workflow systems such as TurKit [44] and CrowdWeaver [31] percolate and amplify these errors across stages of the workflow [43], much like the Game of Telephone.”
HumorTools : A Microtask Workflow for Writing News Satire (2015, methodology): “Various workflows have been proposed to cope with the variability of human output, including iterative improvement [18], findfind-verify[3], and suggestion-and-test[7].”
Challenges in Data Crowdsourcing (2016, background): “There are a variety of systems that provide procedural mechanisms to decompose a problem into tasks issued by humans, including Turkit [30], Automan [12], Dog [7].”
Automated feedback generation for introductory programming assignments (2012, background): “Some courses have experimented with more sophisticated peer evaluation techniques [28] and there is an emerging research area that builds on recent results in crowd-powered systems [7, 30] to provide more structure and better incentives for improving the feedback quality.”
Towards Large-Scale Collaborative Planning: Answering High-Level Search Queries Using Human Computation (2011, methodology): “With the development of TurKit (Little et al. 2010) and other toolkits that provide programmatic access to Amazon's Mechanical Turk (www.mturk.com), researchers are now equipped with the ability to write algorithms (Parameswaran et al. 2011) that involve humans in the loop – to perform basic operati”
Human-in-the-Loop Interpretability Prior (2018, methodology): “Our work resembles human computation algorithms [16] applied to user studies for interpretability as we use the user studies to optimize for interpretability instead of just comparing a model to a baseline.”
A survey of the use of crowdsourcing in software engineering (2017, background): “on integrating the crowd as a part of the program code itself [146], [147], [149]–[151], [220].”
Collaboratively crowdsourcing workflows with turkomatic (2012, methodology): “Using workers to select from redundant answers for quality control is a common strategy in many crowdsourcing applications [6,14].”