TicketTalk: Toward human-level performance with end-to-end, transaction-based dialog systems

By Bill Byrne, K. Krishnamoorthi, Saravanan Ganesh, Mihir Kale. 2020

28 citations recorded by the source corpus.

We present a data-driven, end-to-end approach to transaction-based dialog systems that performs at near-human levels in terms of verbal response quality and factual grounding accuracy. We show that two essential components of the system produce these results: a sufficiently large and diverse, in-domain labeled dataset, and a neural network-based, pre-trained model that generates both verbal responses and API call predictions. In terms of data, we introduce TicketTalk, a movie ticketing dialog dataset with 23,789 annotated conversations. The conversations range from completely open-ended and unrestricted to more structured, both in terms of their knowledge base, discourse features, and number of turns. In qualitative human evaluations, model-generated responses trained on just 10,000 TicketTalk dialogs were rated to “make sense” 86.5% of the time, almost the same as human responses in the same contexts. Our simple, API-focused annotation schema results in a much easier labeling task making it faster and more cost effective. It is also the key component for being able to predict API calls accurately. We handle factual grounding by incorporating API calls in the training data, allowing our model to learn which actions to take and when. Trained on the same 10,000-dialog set, the model’s API call predictions were rated to be correct 93.9% of the time in our evaluations, surpassing the ratings for the corresponding human labels. We show how API prediction and response generation scores improve as the dataset size incrementally increases from 5000 to 21,000 dialogs. Our analysis also clearly illustrates the benefits of pre-training. To facilitate future work on transaction-based dialog systems, we are publicly releasing the TicketTalk dataset at https://git.io/JL8an.

Subfields: Artificial Intelligence, Natural Language Processing, Dialogue & Conversational AI

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Dialogue & Conversational AI

Citations in the corpus

LaMDA: Language Models for Dialog Applications (2022, background): “[42], which fine-tunes the model to use external APIs for movie ticketing dialog.”
Continual Learning in Task-Oriented Dialogue Systems (2020, methodology): “In this paper, we model task-oriented dialogue systems as a seq2seq generation task (Lei et al., 2018; Byrne et al., 2020) that generates both api-calls and responses.”
BiToD: A Bilingual Multi-Domain Dataset For Task-Oriented Dialogue Modeling (2021, background): “[16] Bill Byrne, Karthik Krishnamoorthi, Saravanan Ganesh, and Mihir Sanjay Kale.”
GEMv2: Multilingual NLG Benchmarking in a Single Line of Code (2022, background): “According to Byrne et al. (2021), the Taskmaster-3 (also called TicketTalk) dataset consists of 23,789 movie ticketing dialogs, where the customer’s goal is to purchase tickets after deciding on theater, time, movie name, number of tickets, and date, or opt out of the transaction.”