Evaluating Quality of Chatbots and Intelligent Conversational Agents

By Nicole M. Radziwill, Morgan C. Benton. arXiv.org, 2017

393 citations recorded by the source corpus.

Chatbots are one class of intelligent, conversational software agents activated by natural language input (which can be in the form of text, voice, or both). They provide conversational output in response, and if commanded, can sometimes also execute tasks. Although chatbot technologies have existed since the 1960s and have influenced user interface development in games since the early 1980s, chatbots are now easier to train and implement. This is due to plentiful open source code, widely available development platforms, and implementation options via Software as a Service (SaaS). In addition to enhancing customer experiences and supporting learning, chatbots can also be used to engineer social harm - that is, to spread rumors and misinformation, or attack people for posting their thoughts and opinions online. This paper presents a literature review of quality issues and attributes as they relate to the contemporary issue of chatbot development and implementation. Finally, quality assessment approaches are reviewed, and a quality assessment method based on these attributes and the Analytic Hierarchy Process (AHP) is proposed and examined.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/6db82d07eedd9eb05b2996876486bfb2a141585a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Rediscovering the use of chatbots in education: A systematic literature review (2020, background): “Radziwill and Benton [48], without focusing on any special type of chatbots, argue that the attributes to measure quality should be objective and have a specific category (e.g., performance, functionality, etc.), while also suggesting the concepts of humanity and affection as quality parameters…”
How Should My Chatbot Interact? A Survey on Social Characteristics in Human–Chatbot Interaction Design (2019, background): “[Radziwill and Benton 2017] review the literature on quality issues and attributes for chatbots.”
Artificially intelligent chatbots in digital mental health interventions: a review (2021, background): “ceived as less judgmental, which facilitates self-disclosure among users, and allows for more conversational flexibility [67,84,99].”
An overview of the features of chatbots in mental health: A scoping review (2019, background): “18 language processing and growing availability of large health databases and corpora [51, 52], developers should endeavour to build more artificial intelligence chatbots.”
A Literature Survey of Recent Advances in Chatbots (2021, background): “[24], for example, deﬁne chatbots as a subcategory of conversational agents that are not usually embodied.”
The Personalization of Conversational Agents in Health Care: Systematic Review (2019, background): “feature computer-generated visual virtual characters capable of both verbal and nonverbal communication, and commonly available smart conversational interfaces such as Apple’s Siri, Google’s Google Assistant, Samsung’s Bixby, and Microsoft’s Cortana [1,33,34].”
Artificial Intelligence–Based Chatbots for Promoting Health Behavioral Changes: Systematic Review (2023, neutral): “Third, in 20% (3/15) of studies, the humanistic yet nonhumanistic construct of AI chatbots provided a safe space for the users to discuss, share, and ask for information on sensitive issues [5,22,23,35].”
Conversational agents in healthcare: a systematic review (2018, background): “Advances in machine learning, particularly in neural networks, has allowed for more complex dialogue management methods and more conversational flexibility.(8,9) Given the development of increasingly powerful and connected devices, and growing access to contextual information (such as from sensors),”