Advancing the State of the Art in Open Domain Dialog Systems through the Alexa Prize

By Chandra Khatri, Behnam Hedayatnia, Anu Venkatesh, J. Nunn, Yi Pan, Qing Liu, Han Song, Anna Gottardi, Sanjeev Kwatra, S. Pancholi and 12 more. 2018

87 citations recorded by the source corpus.

Building open domain conversational systems that allow users to have engaging conversations on topics of their choice is a challenging task. Alexa Prize was launched in 2016 to tackle the problem of achieving natural, sustained, coherent and engaging open-domain dialogs. In the second iteration of the competition in 2018, university teams advanced the state of the art by using context in dialog models, leveraging knowledge graphs for language understanding, handling complex utterances, building statistical and hierarchical dialog managers, and leveraging model-driven signals from user responses. The 2018 competition also included the provision of a suite of tools and models to the competitors including the CoBot (conversational bot) toolkit, topic and dialog act detection models, conversation evaluators, and a sensitive content detection model so that the competing teams could focus on building knowledge-rich, coherent and engaging multi-turn dialog systems. This paper outlines the advances developed by the university teams as well as the Alexa Prize team to achieve the common goal of advancing the science of Conversational AI. We address several key open-ended problems such as conversational speech recognition, open domain natural language understanding, commonsense reasoning, statistical dialog management, and dialog evaluation. These collaborative efforts have driven improved experiences by Alexa users to an average rating of 3.61, the median duration of 2 mins 18 seconds, and average turns to 14.6, increases of 14%, 92%, 54% respectively since the launch of the 2018 competition. For conversational speech recognition, we have improved our relative Word Error Rate by 55% and our relative Entity Error Rate by 34% since the launch of the Alexa Prize. Socialbots improved in quality significantly more rapidly in 2018, in part due to the release of the CoBot toolkit.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, background): “Khatri et al. (2018) deﬁne sensitive content more generally as being offensive to people based on gender, demographic factors, culture, or religion.”
Recipes for Safety in Open-domain Chatbots (2020, methodology): “We consider the Wikipedia Toxic Comments dataset (WTC) (Wulczyn et al., 2017) designed to identify personal attacks online, consisting of ∼ 150k examples; we use the version that treats the data as a two-class problem (Khatri et al., 2018a; Dinan et al., 2019c).”
Guiding the Release of Safer E2E Conversational AI through Value Sensitive Design (2022, background): “Existing work on conversational systems often relies on identification of keywords (Ram et al., 2017; Cercas Curry et al., 2018; Fulda et al., 2018; Khatri et al., 2018; Paran-jape et al., 2020), or uses human labels such as flag-ging of a post to train classifiers (Larionov et al., 2018; Cercas…”
True Few-Shot Learning with Language Models (2021, methodology): “We evaluate the generalization loss of the algorithm chosen by CV (likewise for MDL): where Reliability is important for many practical or commercial applications where worst-case performance is important, such as image recognition [35, 36], dialogue systems [37, 38], and robotics [39, 40].”
Neural Generation Meets Real People: Towards Emotionally Engaging Mixed-Initiative Conversations (2020, methodology): “Our system is built on top of the CoBot framework (Khatri et al., 2018).”
SafetyKit: First Aid for Measuring Safety in Open-domain Conversational Systems (2022, background): “Khatri et al. (2018) define sensitive content more generally as offensive to people based on gender, demographic factors, culture, or religion.”
Topical-Chat: Towards Knowledge-Grounded Open-Domain Conversations (2019, background): “Entity Selection: We first selected 300 popular entities spanning 8 topics from a prior human-bot conversational dataset collected during a large-scale open-domain socialbot competition between academic research groups [13].”
Further Advances in Open Domain Dialog Systems in the Third Alexa Prize Socialbot Grand Challenge (2020, methodology): “…scientific advances and reduce time invested into infrastructure, hosting and scaling complex model-driven bots, we created CoBot, a conversational bot toolkit in Python for natural language understanding and dialog management, as described in detail in Khatri et al. (2018) and shown in Figure 1.”
Why So Toxic?: Measuring and Triggering Toxic Behavior in Open-Domain Chatbots (2022, background): “Toxic speech is often related to polarizing topics like gender, politics, and race [26].”