What makes a good conversation? How controllable attributes affect human judgments

By Abigail See, Stephen Roller, Douwe Kiela, Jason Aaron Edward Weston. 2019

327 citations recorded by the source corpus.

A good conversation requires balance – between simplicity and detail; staying on topic and changing it; asking questions and answering them. Although dialogue agents are commonly evaluated via human judgments of overall quality, the relationship between quality and these individual factors is less well-studied. In this work, we examine two controllable neural text generation methods, conditional training and weighted decoding, in order to control four important attributes for chit-chat dialogue: repetition, specificity, response-relatedness and question-asking. We conduct a large-scale human evaluation to measure the effect of these control parameters on multi-turn interactive conversations on the PersonaChat task. We provide a detailed analysis of their relationship to high-level aspects of conversation, and show that by controlling combinations of these variables our models obtain clear improvements in human quality judgments.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Release Strategies and the Social Impacts of Language Models (2019, background): “at Stanford and FAIR demonstrate the potential to improve chatbot performance by optimizing more directly for high-level conversational attributes such as the extent of repetition[58].”
LaMDA: Language Models for Dialog Applications (2022, methodology): “More reliable metrics for dialog modeling require human evaluation [49, 50, 18, 25, 17, 51], as used in this paper.”
Towards a Human-like Open-Domain Chatbot (2020, background): “See et al. (2019) report engagingness is not the same as humanness.”
Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, methodology): “Gehman et al. (2020a) attempt to apply these techniques to toxicity in LLMs. Control techniques have also been employed in dialog, for example, to control for style (Smith et al., 2020b), engagingness (See et al., 2019), or coherence (Xu et al., 2018).”
Recipes for Safety in Open-domain Chatbots (2020, methodology): “See et al. (2019) provides examples of con-trol speciﬁcally aiming at maximizing dialogue engagingness, but does not look at offensiveness.”
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, methodology): “The work of See et al. (2019) goes further and shows that it is possible to control multiple important attributes for chitchat dialogue: repetition, specificity, response-relatedness and question-asking, in order to optimize for well-balanced conversations.”
ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons (2019, methodology): “Both of these baselines exhibit strong repetitive behavior which is known to be highly disfavored by crowd-workers (See et al. 2019).”
Queens Are Powerful Too: Mitigating Gender Bias in Dialogue Generation (2019, methodology): “To create models that can generate a gender-balanced number of gendered words, we propose Conditional Training (CT) for controlling generative model output (Kikuchi et al., 2016; Fan et al., 2017; Oraby et al., 2018; See et al., 2019).”
Plug and Play Language Models: A Simple Approach to Controlled Text Generation (2019, background): “See et al. (2019) note that control with weighted decoding (WD) is difﬁcult and often leads to sacriﬁcing ﬂuency and coherence.”
Human-centric Dialog Training via Offline Reinforcement Learning (2020, methodology): “, 2018), and that reducing repetition and increasing specificity on the part of the model can improve conversation quality (See et al., 2019; Mehri and Eskenazi, 2020).”
Neural Text Generation with Unlikelihood Training (2019, methodology): “Some other lines of work of note are the use of generative control (Kikuchi et al., 2016; Fan et al., 2017; Peng et al., 2018; See et al., 2019) and the retrieve-and-reﬁne setting (Weston et al., 2018; Guu et al., 2018).”
Multi-Dimensional Gender Bias Classification (2020, background): “, 2017), generating questions in chit-chat (See et al., 2019), and reducing bias (Dinan et al.”
DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts (2021, background): “Controlling the output of pretrained language models (LMs) is crucial for achieving useful and safe language generation applications, such as nonoffensive sentence completion or friendly conversation generation (See et al., 2019; Sheng et al., 2020; Gehman et al., 2020).”
The Dialogue Dodecathlon: Open-Domain Knowledge and Image Grounded Conversational Agents (2019, methodology): “Twitter We use a variant of Twitter discussions (text-only), which have been used in many existing studies, e.g. Sordoni et al. (2015); See et al. (2019).”
A Distributional Approach to Controlled Text Generation (2020, background): “These techniques usually referred to as weighted decoding Holtzman et al. (2018); See et al. (2019) this however still requires a heavy search procedure and this biased estimation of sequences that satisfy the global constraint compromises fluency and coherence.”
Neural Generation Meets Real People: Towards Emotionally Engaging Mixed-Initiative Conversations (2020, methodology): “Dialogue acts can support understanding of user intent (Stolcke et al., 2000), and have been successfully employed in previous Alexa Prize socialbots (Yu et al., 2019).”
Controlling Style in Generated Dialogue (2020, background): “…experience when interacting with a model, and provide ways to focus on styles that tend to produce generations with desirable qualities such as being pleasant to talk to, less toxic, more empathetic, or generally better-behaved (See et al., 2019; Roller et al., 2020a; Rashkin et al., 2019).”
Challenges in Building Intelligent Open-domain Dialog Systems (2019, background): “l machine translation benchmarks. Readers refer to [30] for a detailed discussion. These research attempts indicate that what makes a good conversation is a challenging question to answer. See et al. [107] discussed four attributes that are associated with the control of open-domain dialog generatio”
Deploying Lifelong Open-Domain Dialogue Learning (2020, background): “…well as the batch which can help reduce a model’s tendency to repeat itself (Holtzman et al., 2019; Welleck et al., 2020); and (ii) decoding control (See et al., 2019) whereby at decoding time responses are rescaled before scoring based on their speciﬁcity (normalized inverse document frequency).”