The Second Conversational Intelligence Challenge (ConvAI2)

By Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander H. Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur Szlam, Iulian Serban, Ryan J. Lowe and 7 more. 2019

403 citations recorded by the source corpus.

We describe the setting and results of the ConvAI2 NeurIPS competition that aims to further the state-of-the-art in open-domain chatbots. Some key takeaways from the competition are: (1) pretrained Transformer variants are currently the best performing models on this task, (2) but to improve performance on multi-turn conversations with humans, future systems must go beyond single word metrics like perplexity to measure the performance across sequences of utterances (conversations)—in terms of repetition, consistency and balance of dialogue acts (e.g. how many questions asked vs. answered).

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension (2019, methodology): “Dialogue We evaluate dialogue response generation on C ONV AI2 (Dinan et al., 2019 Abstractive QA We use the recently proposed ELI5 dataset to test the model’s ability to generate long free-form answers.”
LaMDA: Language Models for Dialog Applications (2022, methodology): “, [25, 26, 17, 18]), which built on earlier research in neural dialog modeling (e.”
Hi, my name is Martha: Using names to measure and mitigate bias in generative dialogue models (2021, methodology): “…strings; in order to more closely match the BlenderBot setup, we use the ParlAI platform3 to tune DialoGPT on the ConvAI2 dataset, an extension of the PersonaChat dataset containing conversations in which partners speak in the guise of specific personas (Zhang et al., 2018; Dinan et al., 2019).”
Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, methodology): “The BST tasks are based off of the ConvAI2 (Dinan et al., 2020b), Wizard of Wikipedia (Zhang et al., 2018; Dinan et al., 2020b), and Empathetic dialogs (Rashkin et al., 2019) tasks.”
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, methodology): “Thus, approaches that simply optimize perplexity might fail to ask any questions of the user, for example, or can repeat the same topic over and over (Dinan et al., 2020).”
ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons (2019, methodology): “One speaker (termed the Wizard) is meant to be both engaging and knowledgeable on the topics, and has access to an information retrieval system over Wikipedia to supplement their own knowledge.”
Poly-encoders: Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring (2019, methodology): “…over a given input and label candidate, tend to attain much higher accuracies than their counterparts, Bi-encoders (Mazar´e et al., 2018; Dinan et al., 2019), which perform self-attention over the input and candidate label separately and combine them at the end for a ﬁnal…”
Retrieval Augmentation Reduces Hallucination in Conversation (2021, methodology): “Datasets We conduct experiments on two datasets: Wizard of Wikipedia (WoW) (Dinan et al., 2019b) and CMU Document Grounded Conversations (CMU_DoG) (Zhou et al., 2018) which are both sets of knowledge-grounded dialogues collected through human-human crowdworker chats in English, where one of the…”
Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack (2019, background): “The “safe data” is comprised of utterances from the ConvAI2 chit-chat task (Dinan et al., 2019; Zhang et al., 2018) which consists of pairs of humans getting to know each other by discussing their interests.”
Neural Text Generation with Unlikelihood Training (2019, background): “For example, this was observed across all generative models submitted to the ConvAI2 NeurIPS 2018 competition (Dinan et al., 2019).”
Survey of Hallucination in Natural Language Generation (2022, background): “The PersonaChat datasets [31, 228] have accelerated research into persona consistency [68, 91, 126, 208, 219, 224, 232].”
Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learning (2020, methodology): “We conduct our experiment on five diverse datasets covering multiple generation tasks: PersonaChat (Zhang et al., 2018; Dinan et al., 2019) for chit-chat based dialogue (DLG), IWSLT (Cettolo et al.”