SimSensei kiosk: a virtual human interviewer for healthcare decision support

By David DeVault, Ron Artstein, G. Benn, Teresa Dey, Edward Fast, Alesia Gainer, Kallirroi Georgila, Jonathan Gratch, Arno Hartholt, Margot Lhommet and 12 more. Adaptive Agents and Multi-Agent Systems, 2014

730 citations recorded by the source corpus.

We present SimSensei Kiosk, an implemented virtual human interviewer designed to create an engaging face-to-face interaction where the user feels comfortable talking and sharing information. SimSensei Kiosk is also designed to create interactional situations favorable to the automatic assessment of distress indicators, defined as verbal and nonverbal behaviors correlated with depression, anxiety or post-traumatic stress disorder (PTSD). In this paper, we summarize the design methodology, performed over the past two years, which is based on three main development cycles: (1) analysis of face-to-face human interactions to identify potential distress indicators, dialogue policies and virtual human gestures, (2) development and analysis of a Wizard-of-Oz prototype system where two human operators were deciding the spoken and gestural responses, and (3) development of a fully automatic virtual interviewer able to engage users in 15-25 minute interactions. We show the potential of our fully automatic virtual human interviewer in a user study, and situate its performance in relation to the Wizard-of-Oz prototype.

Subfields: Artificial Intelligence, Experimental and Cognitive Psychology, Social Psychology

View Paper: https://www.semanticscholar.org/paper/72f463e6a50bc9738a45aea57062b495331f4031

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Experimental and Cognitive Psychology

In Social Psychology

Citations in the corpus

Neural Approaches to Conversational AI (2018, positive): “n is to enrich the dialogue context. Rather than text-only or speech-only ones, our daily dialogues are often multimodal, and involve both verbal and nonverbal inputs like vision (Bohus et al., 2014; DeVault et al., 2014; de Vries et al., 2017; Zhang et al., 2018a). Challenges such as how to combine”
Why and how to use virtual reality to study human social interaction: The challenges of exploring a new research landscape (2018, methodology): “Second, the USC Institute for Creative Technologies developed the SimSensei Kiosk, Ellie, an autonomous virtual human interviewer able to engage users for a 15–25 min interaction where they would feel comfortable to share personal information (DeVault et al., 2014).”
"I Hear You, I Feel You": Encouraging Deep Self-disclosure through a Chatbot (2020, background): “These results are roughly in line with previous research [41, 12], which indicated that computer agents’ self-disclosure could facilitate their users’ self-disclosure.”
It's only a computer: Virtual humans increase willingness to disclose (2014, methodology): “…have not been directly compared to traditional computer-administered assessments or face-to-face interviews.3 Such comparisons would 3 See DeVault et al. (2014) for initial work in this area. also further explicate the role that rapport plays in how these assessments encourage honest…”
Evaluating Machines by their Real-World Language Use (2020, background): “An AI capable of writing consistently helpful advice – perhaps, a virtual therapist (DeVault et al., 2014) – could greatly help people in need.”
AVEC 2017: Real-life Depression, and Affect Recognition Workshop and Challenge (2017, background): “ese interviews were collected as part of a larger eort to create a computer agent that interviews people and identies verbal and non-verbal indicators of mental illness [6].”
Embodied Conversational Agents in Clinical Psychology: A Scoping Review (2017, positive): “Besides two studies on counseling interventions that also targeted depression [20,48], one study proposed a virtual coach in a CBT-based Web-based platform [57], and one concerned a virtual guide in a serious gaming healing environment [58].”
Advances in natural language processing (2015, background): “They also enable avatars to tutor people in interview or negotiation strategies or to help with health care decisions (18, 19).”
DepAudioNet: An Efficient Deep Model for Audio based Depression Classification (2016, background): “The interviews are collected by a computer agent that interacts with people and identi-ﬁes verbal and non-verbal indicators of mental illness [12].”
Empathy in Virtual Agents and Robots (2017, background): “The SimSensei Kiosk is aimed at addressing problems of depression, anxiety, or post-traumatic stress disorder in patients.”
The Distress Analysis Interview Corpus of human and computer interviews (2014, methodology): “…corpus has been used to support the automated agent’s interactive capabilities by developing custom acoustic and language models for speech recognition, training classifiers for natural language understanding, and informing the creation of dialogue policies; for details, see DeVault et al. (2014).”
Artificial Intelligence (2017, background): “Works like [8,10,22] assume the reciprocal behavior and analyze the development of certain interactions, modeling them as appropriate games.”
Voice Interfaces in Everyday Life (2018, background): “We also note a slight distinction between this interest and virtual or embodied humans/agents, such as SimSensei [5], that are spoken to but also include a visual representation of a human counterpart that audibly and visually responds.”