Towards a Human-like Open-Domain Chatbot

By Daniel De Freitas, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, R. Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu and 1 more. 2020

1,033 citations recorded by the source corpus.

We present Meena, a multi-turn open-domain chatbot trained end-to-end on data mined and filtered from public domain social media conversations. This 2.6B parameter neural network is simply trained to minimize perplexity of the next token. We also propose a human evaluation metric called Sensibleness and Specificity Average (SSA), which captures key elements of a human-like multi-turn conversation. Our experiments show strong correlation between perplexity and SSA. The fact that the best perplexity end-to-end trained Meena scores high on SSA (72% on multi-turn evaluation) suggests that a human-level SSA of 86% is potentially within reach if we can better optimize perplexity. Additionally, the full version of Meena (with a filtering mechanism and tuned decoding) scores 79% SSA, 23% higher in absolute SSA than the existing chatbots we evaluated.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Natural Language Processing

Citations in the corpus

Experience Grounds Language (2020, background): “This is in stark contrast to the disembodied chat bots that are the focus of the current dialogue community (Adiwardana et al., 2020; Zhou et al., 2020; Chen et al., 2018; Serban et al., 2017), which often do not learn from individual experiences and are not given enough of a persistent environment…”
LaMDA: Language Models for Dialog Applications (2022, methodology): “Dialog models [14, 15, 16], one of the most interesting applications of large language models, successfully take advantage of Transformers’ ability to represent long-term dependencies in text [17, 18].”
Self-Consistency Improves Chain of Thought Reasoning in Language Models (2022, methodology): “In addition to sampling, re-ranking is also a common approach to improving generation quality in language models (Adiwardana et al., 2020; Shen et al., 2021).”
Carbon Emissions and Large Neural Network Training (2021, methodology): “We focus on natural language processing (NLP) because it is important in Google products and because of the recent development of many large NLP models, e.g., T5 [Raf19], Meena [Adi20], GShard [Lep20], Switch Transformer [Fed21], and GPT-3 [Bro20].”
OPT: Open Pre-trained Transformer Language Models (2022, background): “Given that LLMs are known to be an integral component of modern dialogue models (Adiwardana et al., 2020; Roller et al., 2021; Thoppilan et al., 2022; Rae et al., 2021; Chowdhery et al., 2022), we additionally evaluate OPT-175B on several open source dialogue datasets.”
The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only (2023, background): “Targeted corpora can also produce domain-specific models (Beltagy et al., 2019), or broaden the expressiveness of models (e.g., for conversational modalities Adiwardana et al. (2020); Thop-pilan et al. (2022)).”
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, neutral): “…re-writing, among others) (Raffel et al., 2020; Zhang et al., 2020b; Aghajanyan et al., 2022; Prabhumoye et al., 2018), (2) or open-ended tasks (such as dialogue generation, prompt-based autocompletion, story generation, and more) (Adiwardana et al., 2020; Brown et al., 2020; Guan et al., 2020).”
“I’m sorry to hear that”: Finding New Biases in Language Models with a Holistic Descriptor Dataset (2022, background): “plained by differences in model size and partially by overall differences in generation between the two classes of models (Adiwardana et al., 2020; Roller et al., 2021; Shuster et al., 2021).”
Recipes for Safety in Open-domain Chatbots (2020, background): “Although we expect these two models to have lower engagingness scores than the BST 2.7B base model, in line with results from Roller et al. (2020); Adiwardana et al. (2020), to our knowledge these methods have not been compared previously in terms of safety evaluations, or the engagingness/safety…”
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, methodology): “(2020)), and recent work has even found perplexity correlates strongly with human evaluations within the same model class (Adiwardana et al., 2020).”
Guiding the Release of Safer E2E Conversational AI through Value Sensitive Design (2022, background): “For example, in a blog post about the ConvAI model Meena (Adiwardana et al., 2020) the authors cite safety challenges as a reason for not releasing the model.”
The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence (2020, methodology): “Meena: Yeah.”
Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completion (2022, background): “BlenderBot 1 was already found to be superior to several other chatbots, in particular Meena (Adiwardana et al., 2020) and DialoGPT (Zhang et al., 2020), and we do not evaluate those here.”
Time-Aware Language Models as Temporal Knowledge Bases (2021, background): “Language models (LMs) have been suggested as repositories of real-world knowledge (Petroni et al., 2019) and there is much interest in using them for tasks such as closed-book question answering (QA; Roberts et al., 2020), fact veriﬁcation (Lee et al., 2020) and dialogue (Adiwardana et al., 2020).”
Retrieval Augmentation Reduces Hallucination in Conversation (2021, background): “Large language models trained on large corpora have made great inroads in the fluency and conversational ability of dialogue agents (Adiwardana et al., 2020; Roller et al., 2020), yielding low perplexity models that have corresponding high token accuracies on in-domain test sets.”
Extracting Training Data from Large Language Models (2020, background): “We thus hope that this publication will spark further discussions on the ethics of memorization and extraction among other companies and research teams that train large LMs [2,36,55,63].”