Spot the Bot: A Robust and Efficient Framework for the Evaluation of Conversational Dialogue Systems

By Jan Deriu, Don Tuggener, Pius von Däniken, Jon Ander Campos, Álvaro Rodrigo, Thiziri Belkacem, Aitor Soroa Etxabe, Eneko Agirre, Mark Cieliebak. 2020

41 citations recorded by the source corpus.

The lack of time-efficient and reliable evaluation methods hamper the development of conversational dialogue systems (chatbots). Evaluations requiring humans to converse with chatbots are time and cost-intensive, put high cognitive demands on the human judges, and yield low-quality results. In this work, we introduce \emph{Spot The Bot}, a cost-efficient and robust evaluation framework that replaces human-bot conversations with conversations between bots. Human judges then only annotate for each entity in a conversation whether they think it is human or not (assuming there are humans participants in these conversations). These annotations then allow us to rank chatbots regarding their ability to mimic the conversational behavior of humans. Since we expect that all bots are eventually recognized as such, we incorporate a metric that measures which chatbot can uphold human-like behavior the longest, i.e., \emph{Survival Analysis}. This metric has the ability to correlate a bot's performance to certain of its characteristics (e.g., \ fluency or sensibleness), yielding interpretable results. The comparably low cost of our framework allows for frequent evaluations of chatbots during their evaluation cycle. We empirically validate our claims by applying \emph{Spot The Bot} to three domains, evaluating several state-of-the-art chatbots, and drawing comparisons to related work. The framework is released as a ready-to-use tool.

Subfields: Artificial Intelligence, Evaluation & Benchmarks, Dialogue & Conversational AI

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

In Dialogue & Conversational AI

Citations in the corpus

Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, background): “…“human-likeness” or “engagingness”, either by automatically comparing with a human ground-truth reference, e.g., by using similarity metrics such as BLEURT (Sellam et al., 2020) or BERTscore (Zhang et al., 2020a), or by asking humans to evaluate this manually (Deriu et al., 2020; Li et al., 2019).”
DynaEval: Unifying Turn and Dialogue Level Evaluation (2021, methodology): “Interactive Evaluation A popular human evaluation method is the interactive evaluation whereby human judges converse with dialogue systems and make the assessment at the end of the conversations (See et al., 2019; Finch and Choi, 2020; Li et al., 2019; Deriu et al., 2020).”
Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agents (2022, background): “Deriu et al. (2020) considers chat logs between pairs of models, again with no humans taking part in the conversations.”
Recent advances in deep learning based dialogue systems: a systematic survey (2021, background): “Deriu et al. (2020) also evaluated the dialogue systems by ranking.”