Improved Deep Learning Baselines for Ubuntu Corpus Dialogs
By Rudolf Kadlec, Martin Schmid, Jan Kleindienst. 2015
114 citations recorded by the source corpus.
This paper presents results of our experiments for the next utterance ranking on the Ubuntu Dialog Corpus -- the largest publicly available multi-turn dialog corpus. First, we use an in-house implementation of previously reported models to do an independent evaluation using the same data. Second, we evaluate the performances of various LSTMs, Bi-LSTMs and CNNs on the dataset. Third, we create an ensemble by averaging predictions of multiple models. The ensemble further improves the performance and it achieves a state-of-the-art result for the next utterance ranking on this dataset. Finally, we discuss our future plans using this corpus.
Subfields: Artificial Intelligence, Information Systems, Dialogue & Conversational AI
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jason Aaron Edward Weston
- Jianfeng Gao
- Mohit Bansal
- Kyunghyun Cho
- Kurt Shuster
- Yoshua Bengio
- Luke S. Zettlemoyer
- Percy Liang
In Information Systems
- Jason Aaron Edward Weston
- Laurent Charlin
- Kurt Shuster
- Iulian Serban
- Joelle Pineau
- Marie-Anne Lachaux
- Samuel Humeau
- Ryan J. Lowe
In Dialogue & Conversational AI
- Jason Aaron Edward Weston
- Kurt Shuster
- Samuel Humeau
- Marie-Anne Lachaux
- Iulian Serban
- Ryan J. Lowe
- Laurent Charlin
- Rui Yan
Citations in the corpus
- A Survey of Available Corpora For Building Data-Driven Dialogue Systems: The Journal Version (2015, methodology): “…using NUC are trained explicitly to maximize the performance on this metric by minimizing the cross-entropy between context-response pairs (Lowe et al., 2015a; Kadlec et al., 2015), the metric could also be used to evaluate a probabilistic generative model trained to output full utterances.”
- Conversational Contextual Cues: The Case of Personalization and History for Response Ranking (2016, background): “Typically, a positive response is mixed with random responses, and then the system is asked to score the right response higher than others (Hu et al., 2014; Kadlec et al., 2015).”
- Enhance word representation for out-of-vocabulary on Ubuntu dialogue corpus (2018, methodology): “Kadlec et al. (2015) built an ensemble of convolution neural network (CNN) (Kim, 2014) and Bi-directional LSTM. Baudiˇs et al. (2016) employed a deep neural network structure (Tan et al., 2015) where CNN was applied to extract features after bi-directional LSTM layer.”
- Real-time Inference in Multi-sentence Tasks with Deep Pretrained Transformers (2019, methodology): “…we consider in this work, several Bi-encoder neural approaches have been considered, in particular Memory Networks (Zhang et al., 2018a) and Transformer Memory networks (Dinan et al., 2019b) as well as LSTMs (Lowe et al., 2015) and CNNs (Kadlec et al., 2015) which encode input and label separately.”
- Modeling Multi-turn Conversation with Deep Utterance Aggregation (2018, methodology): “Our baselines include: • Single-turn matching models : Basic models in (Kadlec et al., 2015; Lowe et al., 2015), including TF-IDF, CNN, RNN, LSTM and biLSTM ; We also explore other advanced single-turn matching models, MV-LSTM (Wan et al., 2016), Match-LSTM (Wang and Jiang, 2015), Attentive-LSTM…”
- A Neural Approach to Automated Essay Scoring (2016, result): “This observation is in line with the findings of some other researchers (Kadlec et al., 2015) and is probably because of the relatively small number of training examples compared to the capacity of the models.”
- Domain Adaptive Training BERT for Response Selection (2019, methodology): “Previous studies of response selection concentrate on recurrent neural model to enhance dialog-response matching (Lowe et al., 2015; Zhou et al., 2016; Wu et al., 2017).”
- Sequential Attention-based Network for Noetic End-to-End Response Selection (2019, methodology): “…and Kleindienst 2015) 0.549 0.684 0.896 0.328 0.515 0.792 LSTM (Kadlec, Schmid, and Kleindienst 2015) 0.638 0.784 0.949 0.365 0.536 0.828 BiLSTM (Kadlec, Schmid, and Kleindienst 2015) 0.630 0.780 0.944 0.355 0.525 0.825 MV-LSTM (Wan et al. 2016) 0.653 0.804 0.946 0.412 0.591 0.857 Match-LSTM…”
- Multi-view Response Selection for Human-Computer Conversation (2016, background): “The response is selected based on the similarity of these two embed-dings (Lowe et al., 2015; Kadlec et al., 2015).”
- Learning to Rank Question-Answer Pairs Using Hierarchical Recurrent Encoder with Latent Topic Clustering (2017, methodology): “(Kadlec, Schmid, and Kleindienst 2015) used convolution neural network (CNN) architecture to embed the sentence while a final output vector was compared to the target text to calculate the matching score.”
- Sequential Match Network: A New Architecture for Multi-turn Response Selection in Retrieval-based Chatbots (2016, methodology): “We considered the following baselines: Basic models : models in (Lowe et al., 2015) and (Kadlec et al., 2015) including TF-IDF, RNN, CNN, LSTM and BiLSTM.”
- LSDSCC: a Large Scale Domain-Specific Conversational Corpus for Response Generation with Diversity Oriented Evaluation Metrics (2018, methodology): “On the other hand, a large number of noises, typos, and slangs distribute in existing large-scale datasets, such as Twitter corpus (Ritter et al., 2011) and Ubuntu dialog corpus (Kadlec et al., 2015).”