Towards Empathetic Open-domain Conversation Models: A New Benchmark and Dataset

By Hannah Rashkin, Eric Michael Smith, Margaret Li, Y-Lan Boureau. 2018

1,186 citations recorded by the source corpus.

One challenge for dialogue agents is recognizing feelings in the conversation partner and replying accordingly, a key communicative skill. While it is straightforward for humans to recognize and acknowledge others’ feelings in a conversation, this is a significant challenge for AI systems due to the paucity of suitable publicly-available datasets for training and evaluation. This work proposes a new benchmark for empathetic dialogue generation and EmpatheticDialogues, a novel dataset of 25k conversations grounded in emotional situations. Our experiments indicate that dialogue models that use our dataset are perceived to be more empathetic by human evaluators, compared to models merely trained on large-scale Internet conversation data. We also present empirical comparisons of dialogue model adaptations for empathetic responding, leveraging existing models or datasets without requiring lengthy re-training of the full model.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Experience Grounds Language (2020, background): “, 2017; Liu and Chai, 2015), or providing emotional support (Rashkin et al., 2019).”
OPT: Open Pre-trained Transformer Language Models (2022, methodology): “, 2019b), Empathetic Dialogues (Rashkin et al., 2019), and Blended Skill Talk (Smith et al.”
“I’m sorry to hear that”: Finding New Biases in Language Models with a Holistic Descriptor Dataset (2022, background): “…purpose-built dialogue datasets, including ones designed to teach consistent personas, knowledge, and empathy (Zhang et al., 2018; Dinan et al., 2018; Rashkin et al., 2019; Smith et al., 2020b; Roller et al., 2021), recall of past conversation details across multiple sessions (Xu et al., 2021a),…”
Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, methodology): “, 2020b), and Empathetic dialogs (Rashkin et al., 2019) tasks.”
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, methodology): “This is a continuation of the approach taken in multiple papers using role-played situations (Rashkin et al., 2019), assigned traits (Shuster et al., 2018, 2019), assigned movie preferences (Kang et al., 2019), or even an entire fantasy universe (Urbanek et al., 2019; Prabhumoye et al., 2020).”
Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completion (2022, methodology): “Finally, we can employ conventional dialogue tasks in this setting as well – PersonaChat (Zhang et al., 2018), Empathetic Dialogues (ED) (Rashkin et al., 2019) and Blended Skill Talk (BST) (Smith et al., 2020) – by using the same procedure as in (Adolphs et al., 2021): we extract an entity from the…”
Learning from Dialogue after Deployment: Feed Yourself, Chatbot! (2019, background): “…dialogue agent’s models are built on the Transformer architecture (Vaswani et al., 2017), which has been shown to perform well on a variety of NLP tasks (Devlin et al., 2018; Radford et al., 2018), including multiple persona-based chat applications (Shuster et al., 2018a,b; Rashkin et al., 2018).”
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage (2022, methodology): “, 2018) X X X X X X X Empathetic Dialogues (Rashkin et al., 2019) X X X X X X X Blended Skill Talk (Smith et al.”
Internet-Augmented Dialogue Generation (2021, methodology): “, 2018) or Empathetic Dialogues (Rashkin et al., 2019) (see Huang et al.”
Recipes for Building an Open-Domain Chatbot (2020, background): “We also report perplexity both before and after ﬁne-tuning each of these models on the ConvAI2, Wizard of Wikipedia, Empathetic Dialogues, and Blended Skill Talk datasets.”
Using In-Context Learning to Improve Dialogue Safety (2023, background): “, 2019b), Empathetic Dialogues (Rashkin et al., 2019), and Blended Skill Talk (Smith et al.”
Can You Put it All Together: Evaluating Conversational Agents’ Ability to Blend Skills (2020, methodology): “well they can display empathy (Rashkin et al., 2019; Lin et al., 2019) or talk about their personal background (Zhang et al.”
CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP (2021, neutral): “Empathetic Dialog (Rashkin et al. 2019) KILT-Wow (Dinan et al.”
Aligning AI With Shared Human Values (2020, background): “To make algorithms prosocial, researchers have proposed imitating temperamental traits such as empathy (Rashkin et al., 2019; Roller et al., 2020), but these have been limited to specific character traits in particular application areas such as chatbots (Krause et al., 2020).”
DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts (2021, background): “…for their respective downstream generation tasks, including emotion-aware text generation (Ghosh et al., 2017; Ficler and Goldberg, 2017), attribute-aware product review generation (Dong et al., 2017), and friendly or empathetic dialogue response generation (See et al., 2019; Rashkin et al., 2019).”
The Dialogue Dodecathlon: Open-Domain Knowledge and Image Grounded Conversational Agents (2019, background): “Rashkin et al. (2019) constructed a dataset of crowdworker conversations grounded in an emotional situation.”
Approximating Interactive Human Evaluation with Self-Play for Open-Domain Dialog Systems (2019, background): “Towards social and emotional coherence, previous works have investigated various features and loss functions based on emotion [26–30].”
Neural Generation Meets Real People: Towards Emotionally Engaging Mixed-Initiative Conversations (2020, methodology): “, 2019) model finetuned on the EmpatheticDialogues dataset (Rashkin et al., 2019).”