Understanding and predicting user dissatisfaction in a neural generative chatbot

By Abigail See. 2021

33 citations recorded by the source corpus.

Neural generative dialogue agents have shown an increasing ability to hold short chitchat conversations, when evaluated by crowdworkers in controlled settings. However, their performance in real-life deployment – talking to intrinsically-motivated users in noisy environments – is less well-explored. In this paper, we perform a detailed case study of a neural generative model deployed as part of Chirpy Cardinal, an Alexa Prize socialbot. We find that unclear user utterances are a major source of generative errors such as ignoring, hallucination, unclearness and repetition. However, even in unambiguous contexts the model frequently makes reasoning errors. Though users express dissatisfaction in correlation with these errors, certain dissatisfaction types (such as offensiveness and privacy objections) depend on additional factors – such as the user’s personal attitudes, and prior unaddressed dissatisfaction in the conversation. Finally, we show that dissatisfied user utterances can be used as a semi-supervised learning signal to improve the dialogue system. We train a model to predict next-turn dissatisfaction, and show through human evaluation that as a ranking function, it selects higher-quality neural-generated utterances.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labels (2022, background): “Instead, users tend to provide free-form textual feedback on what was wrong in their dialogue response to express dissatisfaction when the bot makes errors (See and Manning, 2021).”
What is wrong with you?: Leveraging User Sentiment for Automatic Dialog Evaluation (2022, background): “See and Manning (2021) have shown that predicting next user satisfaction helps to select more relevant system utterances.”