DialCrowd: A toolkit for easy dialog system assessment
By Kyusong Lee, Tiancheng Zhao, Alan W. Black, Maxine Eskénazi. 2018
16 citations recorded by the source corpus.
When creating a dialog system, developers need to test each version to ensure that it is performing correctly. Recently the trend has been to test on large datasets or to ask many users to try out a system. Crowdsourcing has solved the issue of finding users, but it presents new challenges such as how to use a crowdsourcing platform and what type of test is appropriate. DialCrowd has been designed to make system assessment easier and to ensure the quality of the result. This paper describes DialCrowd, what specific needs it fulfills and how it works. It then relates a test of DialCrowd by a group of dialog system developer.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Michael Heck
- Carel van Niekerk
- Nurul Lubis
- Andrey Malinin
- Milica Gavsi'c
- Christian Geishauser
- Shutong Feng
- Hsien-Chin Lin
Citations in the corpus
- Dynamic Dialogue Policy for Continual Reinforcement Learning (2022, methodology): “The website was build using DialCrowd (Lee et al., 2018) and users were hired using Amazon Mechanical Turk.”
- Domain-independent User Simulation with Transformers for Task-oriented Dialogue Systems (2021, methodology): “Human evaluation is performed via DialCrowd (Lee et al., 2018) 6 Experimental Results”
- Uncertainty Measures in Neural Belief Tracking and the Effects on Dialogue Policy Performance (2021, methodology): “Subjects are recruited through the Amazon Mechanical Turk platform to interact with our systems via the DialCrowd platform (Lee et al., 2018).”