A review of evaluation techniques for social dialogue systems
By Amanda Cercas Curry, Helen Wright Hastie, Verena Rieser. 2017
15 citations recorded by the source corpus.
In contrast with goal-oriented dialogue, social dialogue has no clear measure of task success. Consequently, evaluation of these systems is notoriously hard. In this paper, we review current evaluation methods, focusing on automatic metrics. We conclude that turn-based metrics often ignore the context and do not account for the fact that several replies are valid, while end-of-dialogue rewards are mainly hand-crafted. Both lack grounding in human perceptions.
Subfields: Artificial Intelligence, Natural Language Processing, Dialogue & Conversational AI
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jianfeng Gao
- Maxine Eskénazi
- Sergey Levine
- Joelle Pineau
- Shikib Mehri
- Yoshua Bengio
- Mohit Bansal
- Kyunghyun Cho
In Natural Language Processing
- Maxine Eskénazi
- Shikib Mehri
- Tiancheng Zhao
- Jianfeng Gao
- Joelle Pineau
- Ryan J. Lowe
- Peter Henderson
- Koustuv Sinha
In Dialogue & Conversational AI
- Maxine Eskénazi
- Tiancheng Zhao
- Shikib Mehri
- Ryan J. Lowe
- Joelle Pineau
- Sergey Levine
- Peter Henderson
Citations in the corpus
- Ethical Challenges in Data-Driven Dialogue Systems (2017, background): “However, the subjective nature of dialogue (Curry, Hastie, and Rieser 2017; Liu et al. 2016), and the onset of data-driven components (Vinyals and Le 2015; Serban et al. 2017c), leads to several potential dangers with the widespread use of these systems.”
- Offline RL for Natural Language Generation with Implicit Language Q Learning (2022, background): “The lack of easily configurable task settings and reliable simulated evaluations has arguably down progress in applying sophisticated RL algorithms to language and dialogue tasks [49, 50, 51].”
- Beyond Turing: Intelligent Agents Centered on the User (2019, background): “ is important to insure that modules function properly, authors cannot claim that they have designed a better agent until they interactively test a module with users. Section 4 discusses user issues. Curry et al (2017) review assessment techniques for social dialog systems. As mentioned above, PARAD”