Automatic Learning and Evaluation of User-Centered Objective Functions for Dialogue System Optimisation
By Verena Rieser, Oliver Lemon. 2008
22 citations recorded by the source corpus.
The ultimate goal when building dialogue systems is to satisfy the needs of real users, but quality assurance for dialogue strategies is a non-trivial problem. The applied evaluation metrics and resulting design principles are often obscure, emerge by trial-and-error, and are highly context dependent. This paper introduces data-driven methods for obtaining reliable objective functions for system design. In particular, we test whether an objective function obtained from Wizard-of-Oz (WOZ) data is a valid estimate of real users preferences. We test this in a test-retest comparison between the model obtained from the WOZ study and the models obtained when testing with real users. We can show that, despite a low fit to the initial data, the objective function obtained from WOZ data makes accurate predictions for automatic dialogue evaluation, and, when automatically optimising a policy using these predictions, the improvement over a strategy simply mimicking the data becomes clear from an error analysis.
Subfields: Natural Language Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Natural Language Processing
- Y-Lan Boureau
- Emily Dinan
- Gavin Abercrombie
- Dirk Hovy
- Shannon L. Spruit
- Verena Rieser
- A. S. Bergman
- Jason Aaron Edward Weston
Citations in the corpus
- Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, background): “On the other hand, there is a long tradition of “reference free” metrics which estimate the overall quality of a conversation from observable dialog behavior, e.g. (Walker et al., 1997; Rieser & Lemon, 2008; Mehri & Eskenazi, 2020).”