Conversational AI: The Science Behind the Alexa Prize
By Ashwin Ram, Rohit Prasad, Chandra Khatri, Anu Venkatesh, Raefer Gabriel, Qing Liu, J. Nunn, Behnam Hedayatnia, Ming Cheng, Ashish Nagar and 8 more. 2018
301 citations recorded by the source corpus.
Conversational agents are exploding in popularity. However, much work remains in the area of social conversation as well as free-form conversation over a broad range of domains and topics. To advance the state of the art in conversational AI, Amazon launched the Alexa Prize, a 2.5-million-dollar university competition where sixteen selected university teams were challenged to build conversational agents, known as socialbots, to converse coherently and engagingly with humans on popular topics such as Sports, Politics, Entertainment, Fashion and Technology for 20 minutes. The Alexa Prize offers the academic community a unique opportunity to perform research with a live system used by millions of users. The competition provided university teams with real user conversational data at scale, along with the user-provided ratings and feedback augmented with annotations by the Alexa team. This enabled teams to effectively iterate and make improvements throughout the competition while being evaluated in real-time through live user interactions. To build their socialbots, university teams combined state-of-the-art techniques with novel strategies in the areas of Natural Language Understanding, Context Modeling, Dialog Management, Response Generation, and Knowledge Acquisition. To support the efforts of participating teams, the Alexa Prize team made significant scientific and engineering investments to build and improve Conversational Speech Recognition, Topic Tracking, Dialog Evaluation, Voice User Experience, and tools for traffic management and scalability. This paper outlines the advances created by the university teams as well as the Alexa Prize team to achieve the common goal of solving the problem of Conversational AI.
Subfields: Artificial Intelligence, Natural Language Processing, LLM Agents & Tool Use
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jason Aaron Edward Weston
- Jianfeng Gao
- Mohit Bansal
- Yejin Choi
- Minlie Huang
- Percy Liang
- Noah A. Smith
- Luke S. Zettlemoyer
In Natural Language Processing
- Jason Aaron Edward Weston
- Jianfeng Gao
- Yejin Choi
- Y-Lan Boureau
- Emily Dinan
- Minlie Huang
- Kurt Shuster
- Mohit Bansal
In LLM Agents & Tool Use
- Jianfeng Gao
- Maxine Eskénazi
- Jason Aaron Edward Weston
- Tiancheng Zhao
- Mitesh M. Khapra
- Joelle Pineau
- Erik Cambria
- Tom Young
Citations in the corpus
- Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, background): “Offensive system responses For offensive content generated by the systems themselves, Ram et al. (2017) use keyword matching and machine learning methods to detect system responses that are profane, sexual, racially inflammatory, other hate speech, or violent.”
- Recipes for Safety in Open-domain Chatbots (2020, background): “…flagged responses along 5 axes: 1) profane content, 2) sexual content, 3) racially inflammatory content, 4) other hate speech, and 5) violent content (Ram et al., 2017) and define sensitive content as including racism, profanity, hate speech, violence, sexual content or any kind of inappropriate…”
- Towards Exploiting Background Knowledge for Building Conversation Systems (2018, background): “We refer the reader to (Ram et al., 2017) for detailed analysis of these systems.”
- BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage (2022, background): “, 2020) or data (Ram et al., 2018) from several other products have not been openly released.”
- Neural Approaches to Conversational AI (2018, neutral): “201807/rgyKfVsEUdK1BpXf.html 64 specific characters. While Replika is a company, the Replika system has been open-sourced7 and can thus be used as a benchmark for future research. Alexa Prize systems (Ram et al., 2018) are social bots that are exposed to real users, and as such anyone with an Alexa d”
- Can You Put it All Together: Evaluating Conversational Agents’ Ability to Blend Skills (2020, background): “While most commercial dialogue systems rely on hand-coded narrow skills (e.g., see Zhou et al. (2020); Ram et al. (2018)), typically focusing on separate task-oriented features such as alarm setting, calendar entries, etc., we are interested in models that display various qualities in open-domain…”
- MultiWOZ - A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling (2018, background): “As the need of hands-free use cases continues to grow, building a conversational agent that can handle tasks across different application domains has become more and more prominent (Ram et al., 2018).”
- Hurdles to Progress in Long-form Question Answering (2021, neutral): “Another promising direction is extrinsic evaluation (Celikyilmaz et al., 2020) where humans actually interact with systems in real-world scenarios such as the Alexa Prize (Ram et al., 2018) or STORIUM (Akoury et al., 2020).”
- Revealing Persona Biases in Dialogue Systems (2021, neutral): “Such dialogue systems directly interact with millions of end users (Ram et al. 2018).”
- Challenges in Building Intelligent Open-domain Dialog Systems (2019, methodology): “ance the research and development of social bots that are able to converse coherently and engagingly with humans on popular topics such as sports, politics, and entertainment, for at least 20 minutes [71] 1 . The evaluation metric, inspired by the Turing Test [92], is designed to test the social bot”
- Deploying Lifelong Open-Domain Dialogue Learning (2020, background): “A near-exception is the Alexa challenge (Ram et al., 2018) which allows university-based researchers access to a commercial user-base for the span of the competition, however, the data and models are also not released to the rest of the research community.”
- Low-Resource Knowledge-Grounded Dialogue Generation (2020, background): “…domain dialogue systems, due to the applications on social chatbots such as Microsoft XiaoIce (Shum et al., 2018) and virtual assistants such as Amazon Alexa (Ram et al., 2018), have drawn increasing attention from the research community of natural language processing and artificial intelligence.”