Auditing Algorithms : Research Methods for Detecting Discrimination on Internet Platforms

By Christian Sandvig, Kevin Hamilton, K. G. Karahalios, Cédric Langbort. 2014

613 citations recorded by the source corpus.

Subfields: AI Ethics, Fairness & Society

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In AI Ethics, Fairness & Society

Citations in the corpus

How the machine ‘thinks’: Understanding opacity in machine learning algorithms (2016, background): “Sandvig notes that this ‘game of cat-and-mouse’ makes it entirely unlikely that most algorithms will be (or necessarily should be) disclosed to the general public (Sandvig et al., 2014: 9).”
Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019, background): “…board, such as “the need for increasing awareness for what it is actually analyzed,” such as data and phenomena (Ruths and Pfeﬀer, 2014), or the need to understand various dimensions of the automated behavior of platform speciﬁc mechanisms such as their design and algorithms (Sandvig et al., 2014).”
The ethics of algorithms: Mapping the debate (2016, methodology): “Further harm detection mechanisms are necessitated by the capacity of algorithms to disadvantage users in indirect and non-obvious ways that exceed legal deﬁnitions of discrimination (Sandvig et al., 2014; Tufekci, 2015).”
Explaining Explanations in AI (2018, background): “…we need to determine what sort of records algorithmic systems must retain in order to allow for contesting and post-hoc auditing of abnormal events (Mittelstadt, 2016; Sandvig et al., 2014), for instance through identification of classification errors or inaccurate input data (Poulin et al., 2006).”
Quantifying Search Bias: Investigating Sources of Bias for Political Searches in Social Media (2017, background): “These issues have lead researchers, organizations and even governments towards a new avenue of research called “auditing algorithms”, which endeavors to understand if and how an algorithmic system can cause biases, particularly when they are misleading or discriminatory to users [10, 34].”
Auditing Search Engines for Differential Satisfaction Across Demographics (2017, methodology): “Other work sought to detect unfairness through reverse A/B testing with synthetic users [31, 12], algorithmic auditing [37], and analysis of predictions made by supervised machine learning methods [24].”
A Human-Centered Approach to Algorithmic Services: Considerations for Fair and Motivating Smart Community Service Management that Allocates Donations to Non-Profit Organizations (2017, background): “Social implications of algorithmic technologies As the use of algorithms is becoming more prevalent in many sectors of society, both scholarly and industry efforts are increasingly being made to understand the social implications of algorithmic technologies [11, 17, 19, 31] and how these algorithmic”
EU regulations on algorithmic decision-making and a "right to explanation" (2016, methodology): “…studies have attempted to tackle the issue of discrimination within algorithms by introducing tools to both identify (Berendt & Preibusch, 2012; Sandvig et al., 2014) and rectify (Calders & Verwer, 2010; Hajian et al., 2011; Zliobaite et al., 2011; Berendt & Preibusch, 2014; Dive & Khedkar,…”
Algorithmic Folk Theories and Identity: How TikTok Users Co-Produce Knowledge of Identity and Engage in Algorithmic Resistance (2021, background): “Additionally, it would be valuable for further work to audit [69] TikTok’s algorithm to empirically explore the absence or presence of certain social identities on a user’s FYP in light of the identities they hold, as our research showed that many participants’ belief in this theory was informed by ”
Discrimination through Optimization (2019, background): “Auditing algorithms for fairness Following the growing ubiquity of algorithms in daily life, a community formed around investigating their societal impacts [66].”
Questioning the AI: Informing Design Practices for Explainable AI User Experiences (2020, background): “[4] shows, before this wave of work on ML systems, the HCI community have studied explainable systems in various contexts, most notably context-aware applications [12, 56, 57], recommender systems [42], debugging tools [53] and algorithmic transparency [28, 80].”