Can You Put it All Together: Evaluating Conversational Agents’ Ability to Blend Skills

By Eric Michael Smith, Mary Williamson, Kurt Shuster, Jason Aaron Edward Weston, Y-Lan Boureau. 2020

244 citations recorded by the source corpus.

Being engaging, knowledgeable, and empathetic are all desirable general qualities in a conversational agent. Previous work has introduced tasks and datasets that aim to help agents to learn those qualities in isolation and gauge how well they can express them. But rather than being specialized in one single quality, a good open-domain conversational agent should be able to seamlessly blend them all into one cohesive conversational flow. In this work, we investigate several ways to combine models trained towards isolated capabilities, ranging from simple model aggregation schemes that require minimal additional training, to various forms of multi-task training that encompass several skills at all training stages. We further propose a new dataset, BlendedSkillTalk, to analyze how these capabilities would mesh together in a natural conversation, and compare the performance of different architectures and training schemes. Our experiments show that multi-tasking over several tasks that focus on particular capabilities results in better blended conversation performance compared to models trained on a single skill, and that both unified or two-stage approaches perform well if they are constructed to avoid unwanted bias in skill selection or are fine-tuned on our new task.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

OPT: Open Pre-trained Transformer Language Models (2022, methodology): “In particular, we follow Roller et al. (2021), and evaluate on ConvAI2 (Dinan et al., 2020b), Wizard of Wikipedia (Di-nan et al., 2019b), Empathetic Dialogues (Rashkin et al., 2019), and Blended Skill Talk (Smith et al., 2020).”
“I’m sorry to hear that”: Finding New Biases in Language Models with a Holistic Descriptor Dataset (2022, methodology): “We tag each response with a binary label indicating its level of bias, and we then perform style-controlled generation on those labels so that the model can be prompted to generate responses containing lower amounts of bias (Weston et al., 2018; Smith et al., 2020a).”
Hi, my name is Martha: Using names to measure and mitigate bias in generative dialogue models (2021, methodology): “Each conversation started with a set of persona strings randomly chosen from the set of personas used in the BlendedSkillTalk dataset, which BlenderBot was fine-tuned on (Roller et al., 2020; Smith et al., 2020b).”
Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, methodology): “For this, we take 180 input texts from the Blended Skill Talk (BST) (Smith et al., 2020a) validation set.”
Recipes for Safety in Open-domain Chatbots (2020, methodology): “This labelled data was used in Smith et al. (2020b) to train a classiﬁer that was in turned used to label the BST datasets with styles.”
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, background): “…ParlAI tasks1 for a large set of available corpora), public conversations on internet discussion boards or social media, or acted dialogues from crowdsourced workers (Zhang et al., 2018; Dinan et al., 2019b; Shuster et al., 2018; Rashkin et al., 2019; Shuster et al., 2020; Smith et al., 2020).”
Guiding the Release of Safer E2E Conversational AI through Value Sensitive Design (2022, background): “An important benefit of E2E ConvAI models trained in this paradigm is that they can be adapted to new domains or taught new skills just by fine-tuning a pre-trained model on datasets of interest (e.g. Roller et al., 2020; Smith et al., 2020; Solaimon and Dennison, 2021).”
Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completion (2022, methodology): “Finally, we can employ conventional dialogue tasks in this setting as well – PersonaChat (Zhang et al., 2018), Empathetic Dialogues (ED) (Rashkin et al., 2019) and Blended Skill Talk (BST) (Smith et al., 2020) – by using the same procedure as in (Adolphs et al., 2021): we extract an entity from the ”
When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labels (2022, background): “Existing dialogue models are primarily trained on human-human conversations (Conneau et al., 2019; Baumgartner et al., 2020; Smith et al., 2020).”
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage (2022, methodology): “PC, ED, MSC, BST, WizInt and WoW are used for capturing personality, empathy, long-term memory, blending and knowledge as in BlenderBot 1 and 2.”
Recipes for Building an Open-Domain Chatbot (2020, methodology): “In general this extends beyond Con-vAI2 to many open-domain dialogue datasets, such as daily dialogue and Cornell Movies (He et al., 2019), and also when multi-tasking across many of these datasets, as we also do here (Shuster et al., 2019; Smith et al., 2020).”
Constructing Highly Inductive Contexts for Dialogue Safety through Controllable Reverse Generation (2022, methodology): “single-turn data from BST dataset (Smith et al., 2020) in a 4:1 ratio to ensure the model’s performance on normal contexts.”
Controlling Style in Generated Dialogue (2020, methodology): “For ConvAI2, ED, and WoW, we fine-tune on versions of the datasets to which persona strings (like in ConvAI2) and conversational topics (like in WoW) have been added if they are not already present, as in Smith et al. (2020).”
SaFeRDialogues: Taking Feedback Gracefully after Conversational Safety Failures (2021, methodology): “This overall strategy is similar to previous work aiming to endow models with more empathy or knowledge (Smith et al., 2020; Rashkin et al., 2019).”
Director: Generator-Classifiers For Supervised Language Modeling (2022, positive): “We start with a pre-trained BlenderBot 1 model Roller et al. (2020) and ﬁne-tune it on the Blended Skill Talk (BST) tasks (Smith et al., 2020b).”
SafetyKit: First Aid for Measuring Safety in Open-domain Conversational Systems (2022, methodology): “In addition, neural LM generation is hard to control, although there are some first steps in this direction (Khalifa et al., 2021; Smith et al., 2020b).”
"I'm sorry to hear that": finding bias in language models with a holistic descriptor dataset (2022, methodology): “We use a 3B-parameter Transformer-based style classiﬁer from the controlled generation paper of Smith et al. (2020a) to measure the probability that any generated response can be assigned to any one of 217 different styles.”
ProsocialDialog: A Prosocial Backbone for Conversational Agents (2022, methodology): “To accommodate diverse safe contexts we incorporate conversations from existing dialogue datasets as casual ones and use them in our training, including DailyDialog (Li et al., 2017), Empathet-icDialogues (Rashkin et al., 2019), and Blended-SkillTalk (Smith et al., 2020) (brief description of each d”