MuTual: A Dataset for Multi-Turn Dialogue Reasoning

By Leyang Cui, Yu Wu, Shujie Liu, Yue Zhang, Ming Zhou. 2020

176 citations recorded by the source corpus.

Non-task oriented dialogue systems have achieved great success in recent years due to largely accessible conversation data and the development of deep learning techniques. Given a context, current systems are able to yield a relevant and fluent response, but sometimes make logical mistakes because of weak reasoning capabilities. To facilitate the conversation reasoning research, we introduce MuTual, a novel dataset for Multi-Turn dialogue Reasoning, consisting of 8,860 manually annotated dialogues based on Chinese student English listening comprehension exams. Compared to previous benchmarks for non-task oriented dialogue systems, MuTual is much more challenging since it requires a model that be able to handle various reasoning problems. Empirical results show that state-of-the-art methods only reach 71%, which is far behind human performance of 94%, indicating that there is ample room for improving reasoning ability.

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

Citations in the corpus

Survey of Hallucination in Natural Language Generation (2022, background): “There are some reasoning works in the area of dialogue [26, 57, 198], but few in reducing hallucinations.”
Commonsense-Focused Dialogues for Response Generation: An Empirical Study (2021, methodology): “• MuTual (Cui et al., 2020).”
DialogSum: A Real-Life Scenario Dialogue Summarization Dataset (2021, methodology): “Because of different data processing methods and annotation procedures, original dialogues in DailyDialog, DREAM and MuTual are in different formats.”
Comparing Test Sets with Item Response Theory (2021, background): “This variant of MuTual is built so that we can evaluate if the model can select the safe response when all other options are incorrect.”
Is ChatGPT a General-Purpose Natural Language Processing Task Solver? (2023, background): “, 2019]), dialogue (MuTual [Cui et al., 2020]), summarization (SAMSum [Gliwa et al.”
Think Before You Speak: Using Self-talk to Generate Implicit Commonsense Knowledge for Response Generation (2021, methodology): “, 2018) and reasoning (Cui et al., 2020) within dialog systems.”
Recent advances in deep learning based dialogue systems: a systematic survey (2021, methodology): “Dhingra et al. (2016) presented a soft retrieval mechanism that uses a "soft" posterior distribution over the knowledge base to replace the symbolic queries. They further combined this soft retrieval mechanism into a reinforcement learning framework to achieve complete end-to-end training based on u”
A taxonomy, data set, and benchmark for detecting and classifying malevolent dialogue responses (2021, background): “Contextual information is important for dialogue response classification (Cui et al., 2020).”
RICA: Evaluating Robust Inference Capabilities Based on Commonsense Axioms (2020, background): “, 2020), dialogue systems (Cui et al., 2020), and NLI (Williams et al.”
Natural Language Inference in Context - Investigating Contextual Reasoning over Long Texts (2020, background): “2019) Reading Comprehension 7 Sentence Diverse MuTual (Cui et al. 2020b) Next Utterance Prediction 3 Dialogue Exam ReClor (Yu et al.”
CIDER: Commonsense Inference for Dialogue Explanation and Reasoning (2021, background): “MuTual (Cui et al., 2020) is a manually annotated dataset for multi-turn dialogue reasoning.”
Task-specific Objectives of Pre-trained Language Models for Dialogue Adaptation (2020, methodology): “Response Selection (RS) MuTual (Cui et al. 2020) (Zhao, Lala, and Kawahara 2020) are chosen for turn-level evaluation.”
An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-Tuning (2023, unknown): “Speciﬁcally, instructions for a particular type of task (such as generating headlines) are used to tune the LLMs in each training phase, and the model does not have access to previously learned tasks.”
FCM: A Fine-grained Comparison Model for Multi-turn Dialogue Reasoning (2021, unknown): “Recently, an open domain multi-turn dialogue reasoning dataset (MuTual) (Cui et al., 2020) is proposed to facilitate the reasoning capabilities of conversation models.”
SemEval-2020 Task 4: Commonsense Validation and Explanation (2020, background): “MuTual provides a dataset for Multi-Turn dialogue reasoning in the commonsense area (Cui et al., 2020).”
Think Before You Speak: Explicitly Generating Implicit Commonsense Knowledge for Response Generation (2021, methodology): “, 2019), MuTual (Cui et al., 2020), and SocialIQA-prompted CommonsenseDialogues (Zhou et al.”