Analyzing Assumptions in Conversation Disentanglement Research Through the Lens of a New Dataset and Model

By Jonathan K. Kummerfeld, S. R. Gouravajhala, Joseph Peper, V. Athreya, R. Chulaka Gunasekara, Jatin Ganhotra, Siva Sankalp Patel, Lazaros C. Polymenakos, Walter S. Lasecki. 2018

22 citations recorded by the source corpus.

Disentangling conversations mixed together in a single stream of messages is a difficult task with no large annotated datasets. We created a new dataset that is 25 times the size of any previous publicly available resource, has samples of conversation from 152 points in time across a decade, and is annotated with both threads and a within-thread reply-structure graph. We also developed a new neural network model, which extracts conversation threads substantially more accurately than prior work. Using our annotated data and our model we tested assumptions in prior work, revealing major issues in heuristically constructed resources, and identifying how small datasets have biased our understanding of multi-party multi-conversation chat.

Subfields: Evaluation & Benchmarks, Dialogue & Conversational AI

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

1. Jianfeng Gao
2. Dhruv Batra
3. Luis Fernando D'Haro
4. QUAN LIU
5. Devi Parikh
6. Lazaros C. Polymenakos
7. Zhenhua Ling
8. Chiori Hori

In Dialogue & Conversational AI

1. Jianfeng Gao
2. Michel Galley
3. QUAN LIU
4. Zhenhua Ling
5. Wen Wang
6. Jia-Chen Gu
7. Qian Chen
8. Yu-Ping Ruan

Citations in the corpus

• Building Sequential Inference Models for End-to-End Response Selection (2018, background): “The DSTC7 Track 1 organizers provided two datasets (Kummerfeld et al. 2018).”
• Sequential Attention-based Network for Noetic End-to-End Response Selection (2019, methodology): “Ubuntu dataset The Ubuntu dataset includes two party conversations from Ubuntu Internet Relay Chat (IRC) channel (Kummerfeld et al. 2018).”
• Dialog System Technology Challenge 7 (2019, methodology): “These are extracted using the model described by Kummerfeld et al. [2018], instead of the heuristic approach used in Lowe et al. [2015]. This approach produced 135,000 conversations, which we sample 100,000 of for training and 1,000 for testing.”