Patient and Consumer Safety Risks When Using Conversational Assistants for Medical Information: An Observational Study of Siri, Alexa, and Google Assistant

By Timothy W. Bickmore, Ha Trinh, S. Ólafsson, Teresa K. O'Leary, Reza Asadi, Nathaniel M. Rickles, Ricardo Cruz. 2018

273 citations recorded by the source corpus.

Background Conversational assistants, such as Siri, Alexa, and Google Assistant, are ubiquitous and are beginning to be used as portals for medical services. However, the potential safety issues of using conversational assistants for medical information by patients and consumers are not understood. Objective To determine the prevalence and nature of the harm that could result from patients or consumers using conversational assistants for medical information. Methods Participants were given medical problems to pose to Siri, Alexa, or Google Assistant, and asked to determine an action to take based on information from the system. Assignment of tasks and systems were randomized across participants, and participants queried the conversational assistants in their own words, making as many attempts as needed until they either reported an action to take or gave up. Participant-reported actions for each medical task were rated for patient harm using an Agency for Healthcare Research and Quality harm scale. Results Fifty-four subjects completed the study with a mean age of 42 years (SD 18). Twenty-nine (54%) were female, 31 (57%) Caucasian, and 26 (50%) were college educated. Only 8 (15%) reported using a conversational assistant regularly, while 22 (41%) had never used one, and 24 (44%) had tried one “a few times.“ Forty-four (82%) used computers regularly. Subjects were only able to complete 168 (43%) of their 394 tasks. Of these, 49 (29%) reported actions that could have resulted in some degree of patient harm, including 27 (16%) that could have resulted in death. Conclusions Reliance on conversational assistants for actionable medical information represents a safety risk for patients and consumers. Patients should be cautioned to not use these technologies for answers to medical questions they intend to act on without further consultation from a health care provider.

Subfields: Applied Psychology, Artificial Intelligence, General Health Professions, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In General Health Professions

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “In high-stakes applications like medicine, LMs may misdiagnose, suggest wrong medication dosages (Bickmore et al., 2018), or even suggest suicide (Daws, 2020).”
Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, background): “Sample conversational assistant interactions resulting in potential harm to the user from (Bickmore et al., 2018).”
Taxonomy of Risks posed by Language Models (2022, background): “For example, misin-formation on medical dosages may lead a user to cause harm to themselves [21, 130].”
SafetyKit: First Aid for Measuring Safety in Open-domain Conversational Systems (2022, background): “However, what is an “appropriate” reply in such situations is dependent on the context of deployment (e.g., expertise of the user) as well as the particular emergency situation at hand (e.g., self-harm vs. general medical enquiry cf. Bickmore et al. (2018)), and will benefit from expert guidance.”
The Personalization of Conversational Agents in Health Care: Systematic Review (2019, background): “Using CAs with unconstrained natural language input can be risky [66].”
Large Language Model Alignment: A Survey (2023, background): “This not only affects the trustworthiness of LLMs in general domains, but also limits their applications in professional fields such as medicine (Bickmore et al., 2018) and law (Iu and Wong, 2023).”
SafeText: A Benchmark for Exploring Physical Safety in Language Models (2022, background): “Currently released technology can put users’ physical safety at risk by providing unsafe medical advice (Bickmore et al., 2018) or promoting dangerous viral challenges 1 to unsuspecting users.”
On the Safety of Conversational Models: Taxonomy, Dataset, and Benchmark (2021, background): “Previous works also extended the border of dialogue safety such as patient and consumer safety risks (Bickmore et al., 2018), stereotype (Barikeri et al.”
Conversational Agents in Health Care: Scoping Review and Conceptual Analysis (2020, methodology): “Two studies examined the acceptability of conversational agents for health care service delivery [48,87].”