Why we should have seen that coming: comments on Microsoft's tay "experiment," and wider implications

By Marty J. Wolf, K. Miller, F. Grodzinsky. 2017

218 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Information Systems, Safety Research, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Safety Research

In Safety, Robustness & Interpretability

Citations in the corpus

Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “…context and content control, deployed language generators have quickly derailed to racist, homophobic, hateful comments (Hunt, 2016; Jang, 2021; Wolf et al., 2017; Vincent, 2022), compromised user privacy (Carlini et al., 2021), spread disinformation (Shao et al., 2018), and even encouraged…”
Artificial Intelligence, Values, and Alignment (2020, background): “We have already seen some examples of agent misalignment in the real world, for example with chatbots that ended up promoting abusive content once they were allowed to interact freely with people online (Wolf et al., 2017).”
What’s in the Box? An Analysis of Undesirable Content in the Common Crawl Corpus (2021, background): “Generally speaking, the presence of even a small fraction of websites that incite hate in training corpora is worrisome since it can result in models that replicate this kind of discourse when prompted (Wolf et al., 2017; Carlini et al., 2020).”
Chat as Expected: Learning to Manipulate Black-box Neural Dialogue Models (2020, background): “For example, Tay, an AI chatbot developed by Microsoft, was shut down shortly after release due to its racism, sexist and obscene speech (Wolf et al., 2017).”
The Moral Integrity Corpus: A Benchmark for Ethical Dialogue Systems (2022, unknown): “It is reasonable to expect adversarial probes and strategically-chosen questions (Wolf et al., 2017), which can prompt toxic or immoral behaviors, even in “detoxiﬁed” models that were trained on carefully sanitized inputs (Gehman et al., 2020; Cercas Curry and Rieser, 2018).”
The ethics of ChatGPT - Exploring the ethical issues of an emerging technology (2024, methodology): “This has been a key short-coming of previous models such as Microsoft ’ s chatbot Tay (Wolf et al., 2017).”
ORES (2019, background): “approach that seeks to expand access to a potentially harmful technology should also be mindful about unintended uses and abuses, as adversaries and harms in the content moderation space are numerous [70, 81, 108]. Finally, we also see potential in allowing Wikipedians to freely train, test, and use”
Towards a Standard for Identifying and Managing Bias in Artificial Intelligence (2022, background): “As a result, the model may demonstrate unknown and erratic behavior and challenges for reproducibility and explainability [163].”
Robust Conversational Agents against Imperceptible Toxicity Triggers (2022, background): “We acknowledge that our attack can be used by unethical adversaries to force the models to generate toxic responses which is undesirable as also previously observed in chatbots (Wolf et al., 2017; Henderson et al., 2018; Dinan et al., 2021).”
Learning from data in the mixed adversarial non-adversarial case: Finding the helpers and ignoring the trolls (2022, background): “Hence it appears there may be a signiﬁcant amount of trolls in real world data, as is expected (Wolf et al., 2017).”