Data Collection for Dialogue System: A Startup Perspective
By Yiping Kang, Yunqi Zhang, Jonathan K. Kummerfeld, Lingjia Tang, Jason Mars. 2018
32 citations recorded by the source corpus.
Industrial dialogue systems such as Apple Siri and Google Now rely on large scale diverse and robust training data to enable their sophisticated conversation capability. Crowdsourcing provides a scalable and inexpensive way of data collection but collecting high quality data efficiently requires thoughtful orchestration of the crowdsourcing jobs. Prior study of this topic have focused on tasks only in the academia settings with limited scope or only provide intrinsic dataset analysis, lacking indication on how it affects the trained model performance. In this paper, we present a study of crowdsourcing methods for a user intent classification task in our deployed dialogue system. Our task requires classification of 47 possible user intents and contains many intent pairs with subtle differences. We consider different crowdsourcing job types and job prompts and analyze quantitatively the quality of the collected data and the downstream model performance on a test set of real user queries from production logs. Our observation provides insights into designing efficient crowdsourcing jobs and provide recommendations for future dialogue system data collection process.
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
- Boualem Benatallah
- Mohammad-Ali Yaghoub-Zadeh-Fard
- Stefan Larson
- F. Casati
- Anish Mahendran
- Joseph Peper
- Christopher Clarke
- Andrew Lee
Citations in the corpus
- An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction (2019, methodology): “To collect additional data for each intent, we used the rephrase and scenario crowdsourcing tasks proposed by Kang et al. (2018).”
- User Utterance Acquisition for Training Task-Oriented Bots: A Review of Challenges, Techniques and Opportunities (2020, background): “Coverage (CVG) also measures how well a list of utterances covers the space of all possible utterances for an intent.(8) It is calculated by averaging the similarity score (percentage of n-gram matches) between each utterance in the real user queries (R) and its most similar utterance in the collect”