Why We Should Have Seen That Coming

By K. Miller, Marty J. Wolf, F. Grodzinsky. 2017

69 citations recorded by the source corpus.

In this paper we examine the case of Tay, the Microsoft AI chatbot that was launched in March, 2016. After less than 24 hours, Microsoft shut down the experiment because the chatbot was generating tweets that were judged to be inappropriate since they included racist, sexist, and anti-Semitic language. We contend that the case of Tay illustrates a problem with the very nature of learning software (LS is a term that describes any software that changes its program in response to its interactions) that interacts directly with the public, and the developer’s role and responsibility associated with it. We make the case that when LS interacts directly with people or indirectly via social media, the developer has additional ethical responsibilities beyond those of standard software. There is an additional burden of care.

Subfields: Artificial Intelligence, Information Systems, Safety Research, Time-Series & Forecasting

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Safety Research

In Time-Series & Forecasting

Citations in the corpus

Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, background): ““Tay” refers to the Microsoft AI chatbot, which was launched and subsequently shut down for producing offensive language in March 2016 (Miller et al., 2017).”
Recipes for Safety in Open-domain Chatbots (2020, methodology): “To aid this study we have investigation techniques of crowdsourcing safety evaluations, and built an adversarially created dialogue safety training and evaluation set, which we will publicly release, along with our models in ParlAI 7 .”
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, methodology): “While engagingness is necessary for people to be willing to talk to a conversational agent, it is not sufficient: Tay (Neff and Nagy, 2016; Miller, Wolf, and Grodzinsky, 2017) is an example of agent that might have been engaging, but in a way that required its removal.”
Guiding the Release of Safer E2E Conversational AI through Value Sensitive Design (2022, background): “Generating this type of content can cause harm to users, and poses a reputational risk to the organisation releasing the model, for instance when the bot voices undesirable or controversial opinions, e.g., Tay’s anti-semitic stances (Miller et al., 2017).”
Bot-Adversarial Dialogue for Safe Conversational Agents (2021, methodology): “We consider the same architecture and setup as in BlenderBot (Roller et al., 2020), which employs a Seq2Seq Trans-former architecture (Vaswani et al., 2017), with an implementation based on the ParlAI version (Miller et al., 2017a).”
SafetyKit: First Aid for Measuring Safety in Open-domain Conversational Systems (2022, background): “In addition, Miller et al. (2017) argue that adversarial attacks must be expected and planned for when designing such systems based on their analysis of Tay (see §1).”
Vision, challenges, roles and research issues of Artificial Intelligence in Education (2020, methodology): “For example, a smart tutee could be a chatbot such as Microsoft Tay (Wolf, Miller, & Grodzinsky, 2017) with a natural language processing interface and arti ﬁ cial neural networks.”