Turkers, Scholars, "Arafat" and "Peace": Cultural Communities and Algorithmic Gold Standards

By Shilad Sen, Margaret E. Giesel, Rebecca Gold, Benjamin M Hillmann, Matt Lesicko, Samuel Naden, Jesse Russell, Zixiao Wang, Brent J. Hecht. 2015

68 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Communication, Computer Science Applications, AI Ethics, Fairness & Society, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. A. Hanna
2. Emily L. Denton
3. Inioluwa Deborah Raji
4. Amandalynne Paullada
5. Carlos Castillo
6. Emily M. Bender
7. Fernando Diaz
8. Emre Kıcıman

In Communication

1. Alexandra Olteanu
2. Fernando Diaz
3. Carlos Castillo
4. Emre Kıcıman
5. Emily M. Bender
6. Emily L. Denton
7. Inioluwa Deborah Raji
8. Muhammad Imran

In Computer Science Applications

1. Emily L. Denton
2. Alexandra Olteanu
3. Carlos Castillo
4. A. Hanna
5. Fernando Diaz
6. Emre Kıcıman
7. Os Keyes
8. Inioluwa Deborah Raji

In AI Ethics, Fairness & Society

1. Alexandra Olteanu
2. Fernando Diaz
3. Carlos Castillo
4. Emily M. Bender
5. Emre Kıcıman
6. Emily L. Denton
7. A. Hanna
8. Os Keyes

In Human-Computer Interaction

1. Emily L. Denton
2. Emre Kıcıman
3. A. Hanna
4. Os Keyes
5. Inioluwa Deborah Raji

Citations in the corpus

• CrowdWorkSheets: Accounting for Individual and Collective Identities Underlying Crowdsourced Dataset Annotation (2022, neutral): “Since annotators from different communities can produce significantly different annotations given the same task [53], it is important to recognize that annotator selection may have a significant impact on the labels of your dataset.”
• Data and its (dis)contents: A survey of dataset development and use in machine learning research (2020, background): “Sen et al.(58) have questioned the legitimacy of the notion of a gold standard dataset for certain tasks, empirically demon-”
• Do Datasets Have Politics? Disciplinary Values in Computer Vision Dataset Development (2021, neutral): “At the level of data classifications more broadly, Sen et al. problematize the broad use of so-called “universal gold standards” in benchmark datasets at all [103].”
• Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019, background): “However, some “gold standards” have been found to vary depending on who is doing the annotation, and this, in turn, may affect the algorithmic performance (Sen et al., 2015).”
• The Misgendering Machines (2018, background): “the costs and risks of assuming that an algorithm designed based on a constrained population or framing of a problem will adequately represent reality [92].”
• Tools and methods in participatory modeling: Selecting the right tool for the job (2018, background): “The data may differ significantly from the judgment of the experts, especially when more expertise is required (Sen et al., 2015).”