Disembodied Machine Learning: On the Illusion of Objectivity in NLP

By Zeerak Talat, Joachim Bingel, Isabelle Augenstein. 2021

57 citations recorded by the source corpus.

Machine Learning seeks to identify and encode bodies of knowledge within provided datasets. However, data encodes subjective content, which determines the possible outcomes of the models trained on it. Because such subjectivity enables marginalisation of parts of society, it is termed (social) `bias' and sought to be removed. In this paper, we contextualise this discourse of bias in the ML community against the subjective choices in the development process. Through a consideration of how choices in data and model development construct subjectivity, or biases that are represented in a model, we argue that addressing and mitigating biases is near-impossible. This is because both data and ML models are objects for which meaning is made in each step of the development pipeline, from data selection over annotation to model training and analysis. Accordingly, we find the prevalent discourse of bias limiting in its ability to address social marginalisation. We recommend to be conscientious of this, and to accept that de-biasing methods only correct for a fraction of biases.

Subfields: Artificial Intelligence, Safety Research, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety Research

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

AI and the Everything in the Whole Wide World Benchmark (2021, background): “This very speciﬁc subset of American English text in the presence of annotation artifacts [Gururangan et al., 2018] results in inherently subjective outcomes [Waseem et al., 2020].”
Evaluating the Social Impact of Generative AI Systems in Systems and Society (2023, background): “” Biases are not only introduced in the data pipeline but throughout the entire machine learning pipeline [237].”
MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, background): “[117] Zeerak Waseem, Smarika Lulz, Joachim Bingel, and Isabelle Augenstein.”
MERLOT: Multimodal Neural Script Knowledge Models (2021, background): “Objectives like the common masked language modeling objective reinforce this, as the model must predict everything that was ever said – the equivalent of a ‘view from nowhere’ [34, 101].”
Data Governance in the Age of Large-Scale Data-Driven Language Technology (2022, background): “[130] Zeerak Talat, Smarika Lulz, Joachim Bingel, and Isabelle Augenstein.”
You reap what you sow: On the Challenges of Bias Evaluation Under Multilingual Settings (2022, methodology): “…(Gehman et al., 2020; Lin et al., 2021) and demograph-028 ics (Sheng et al., 2019; Liang et al., 2021; Dev et al., 2021a)—in part due to the training data used (Dunn, 2020; Gao et al., 2020; Bender et al., 2021), and the design of modeling processes (Talat et al., 2021; Hovy and Prabhumoye, 2021).”
Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection (2021, background): “, contains hate speech, abuse, or is offensive) is inherently a subjective task that requires a nuanced understanding of the pragmatic implications of language (Fiske, 1993; Croom, 2011; Waseem et al., 2021).”
A Survey of Race, Racism, and Anti-Racism in NLP (2021, background): “The representativeness of data sets is a well-discussed issue in social-oriented tasks, like inferring public opinion (Olteanu et al., 2019), but this issue is also an important considera-5 tion in ‘neutral’ tasks like parsing (Waseem et al., 2021).”
Gender Bias in Machine Translation (2021, background): “, 2019), and considering everyone’s subjective positionality—and, crucially, also the lack of diversity within technology teams (Schluter, 2018; Waseem et al., 2020).”
Reusable Templates and Guides For Documenting Datasets and Models for Natural Language Processing and Generation: A Case Study of the HuggingFace and GEM Data and Model Cards (2021, background): “As discussed by Waseem et al. (2021), datasets are the result of subjective choices made by the dataset creators.”
On the Machine Learning of Ethical Judgments from Natural Language (2022, methodology): “…through analysis of the models, their training regimes, and the data that they rely on (Hall Maud-slay et al., 2019; Zhao et al., 2019; Dinan et al., 2020a,b; Vargas and Cotterell, 2020; Smith and Williams, 2021; Talat et al., 2021), rather than seeking to imbue models with a sense of ethics.”
Back to the Future: On Potential Histories in NLP (2022, background): “Data-making has been conceptualized as ﬁction (e.g. Gitelman, 2013) and ML researchers have also begun to conceptualize ML, and data, as subjective (Talat et al., 2021) and value-laden (Birhane et al., 2022).”