Turkopticon: interrupting worker invisibility in amazon mechanical turk

By Lilly C. Irani, Michael Six Silberman. 2013

897 citations recorded by the source corpus.

Subfields: Computer Science Applications, Safety Research, AI Ethics, Fairness & Society, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Science Applications

In Safety Research

In AI Ethics, Fairness & Society

In Human-Computer Interaction

Citations in the corpus

CrowdWorkSheets: Accounting for Individual and Collective Identities Underlying Crowdsourced Dataset Annotation (2022, neutral): “Yet, some platforms give requesters full rights over the data they receive, regardless of whether they accept or reject it, and workers have no way of taking legal action if requesters use rejected work anyway [30]; Roberts [43] describes this system as one that “enables wage theft”.”
Data and its (dis)contents: A survey of dataset development and use in machine learning research (2020, background): “crowdworkers from the dataset developers constructing annotation tasks, rendering both the workers and the labor concerns they might face invisible.(124,125) Such labor concerns include low and unstable wages, unfair treatment by task requesters, and barriers to worker solidarity and collective acti”
Do Datasets Have Politics? Disciplinary Values in Computer Vision Dataset Development (2021, neutral): “…datasets, including descriptions of data annotators [49, 61], data availability for research purposes [12, 83], properties of data subjects and categories [98], and descriptions of decisions more generally which typically are obviated or obscured in the data collection process [10, 43, 44].”
Towards Accountability for Machine Learning Datasets: Practices from Software Engineering and Infrastructure (2020, background): “ring for datasets (including the collection, cleaning, and annotation of data) receives few, if any, accolades in the mainstream field. Since it is often an unpaid burden on top of existing workloads [73, 124, 138], careful dataset work is unlikely to occur at all. Publications focusing solely on da”
Bringing the People Back In: Contesting Benchmark Machine Learning Datasets (2020, background): “…systems has focused on the importance of analyzing interrogating unspoken work practices of technical experts (Passi & Barocas, 2019; Seaver, 2019), subject matter experts in acts of data repair (Sachs, 2019), and crowd laborers (Irani & Silberman, 2013; Salehi et al., 2015; Gray & Suri, 2019).”
On the genealogy of machine learning datasets: A critical history of ImageNet (2021, background): “, 2020), and data labor often goes undocumented (Irani and Silberman, 2013; Gray and Suri, 2019; Miceli et al., 2020).”
Power to the People? Opportunities and Challenges for Participatory AI (2022, background): “Participation has also been instrumental in designing and building algorithms that serve communities for purposes such as operating on-demand food donation transportation service [54] as well as for building tools, for instance, (Turkopticon) that allow low wage workers — Amazon Mechanical Turkers, ”
Ethical norms and issues in crowdsourcing practices: A Habermasian analysis (2018, background): “Notably, these include studies based on a view of (microwork) crowdsourcing as ethically problematic (Irani and Silberman 2014; Silberman et al. 2010), with Irani and colleagues providing tools such as Turkopticon (Irani and Silberman 2013) and Dynamo (Salehi et al. 2015) to support crowd workers.”
Conducting Clinical Research Using Crowdsourced Convenience Samples. (2016, background): “Requesters’ payment decisions also directly influence worker reputation and access to work, whereas workers are forced to informally track and report on requester reputation in forums outside of MTurk (e.g., Turkopticon; Irani & Silberman 2013).”
“Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI (2021, background): “Learning from HCI scholarship on ways to recognise the human labour in preparing, curating, and nurturing data that powers AI models [34, 117], among crowd workers [34, 57, 79], office clerks [85], and health workers [58] can be helpful.”
The Expertise Involved in Deciding which HITs are Worth Doing on Amazon Mechanical Turk (2021, methodology): “The design of the AMT platform itself has led to several power asymmetries between Turkers and requesters, and has promoted the invisibility and commoditization of work completed on AMT [1, 3, 28, 29, 38, 46].”
Working with Machines: The Impact of Algorithmic and Data-Driven Management on Human Workers (2015, background): “Drivers used online forums to post about bad passengers so that other drivers could avoid them, similar to self-regulation strategies of mTurkers [17].”
What Do Platforms Do? Understanding the Gig Economy (2020, background): “Irani (2015, p. 231) found that programmers employ macrotaskers on AMT in ways that “hide their labor force,” enabling them to frame their startups as creative technology companies even as they rely on an army of crowdworkers performing mundane tasks (see also Shestakofsky 2017).”
The rise of the "just-in-time workforce": on-demand work, crowdwork and labour protection in the "gig-economy" (2015, background): “10 Conditions of Work and Employment Series No. 71 Nonetheless, some online platforms have appeared that try to organise crowdworkers around common objectives taking into account the singularity of this form of work and of the needs related thereto (Irani and Silberman, 2013).”