Frames: a corpus for adding memory to goal-oriented dialogue systems
By Layla El Asri, Hannes Schulz, Shikhar Sharma, Jeremie Zumer, Justin Harris, Emery Fine, Kaheer Suleman. 2017
280 citations recorded by the source corpus.
This paper proposes a new dataset, Frames, composed of 1369 human-human dialogues with an average of 15 turns per dialogue. This corpus contains goal-oriented dialogues between users who are given some constraints to book a trip and assistants who search a database to find appropriate trips. The users exhibit complex decision-making behaviour which involve comparing trips, exploring different options, and selecting among the trips that were discussed during the dialogue. To drive research on dialogue systems towards handling such behaviour, we have annotated and released the dataset and we propose in this paper a task called frame tracking. This task consists of keeping track of different semantic frames throughout each dialogue. We propose a rule-based baseline and analyse the frame tracking task through this baseline.
Subfields: Artificial Intelligence, Natural Language Processing, Dialogue & Conversational AI
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jianfeng Gao
- Jason Aaron Edward Weston
- Mohit Bansal
- Percy Liang
- Caiming Xiong
- Richard Socher
- Yoshua Bengio
- Luke S. Zettlemoyer
In Natural Language Processing
- Jason Aaron Edward Weston
- Jianfeng Gao
- Mohit Bansal
- Richard Socher
- Caiming Xiong
- Kurt Shuster
- Emily Dinan
- Douwe Kiela
In Dialogue & Conversational AI
- Kurt Shuster
- Jason Aaron Edward Weston
- Arsha Nagrani
- Cordelia Schmid
- Paul Hongsuck Seo
- Pei-hao Su
- Milica Gasic
- Eric Michael Smith
Citations in the corpus
- Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, background): “Traditional goal-oriented dialogue has focused on narrow tasks that would typically be useful for a dialogue-based assistant, for example restaurant (Henderson, Thomson, and Williams, 2014), taxi, train, and hotel (Budzianowski et al., 2018) or trip (El Asri et al., 2017) booking.”
- ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons (2019, background): “Goal-oriented tasks typically have a clearer evaluation, e.g. task completion can be measured if the correct actions are taken (Hastie 2012; Henderson, Thomson, and Williams 2014; Bordes, Boureau, and Weston 2017; El Asri et al. 2017; Wen et al. 2017).”
- Wizard of Wikipedia: Knowledge-Powered Conversational agents (2018, background): “In the area of goal-directed dialogue, separate from open domain chit-chat, such as airline (El Asri et al., 2017) or restaurant booking (Henderson et al., 2014; Bordes et al., 2017), knowledge conditioning is typically employed by allowing access to a database through API calls or otherwise.”
- What makes a good conversation? How controllable attributes affect human judgments (2019, background): “Dialogue Dialogue evaluation is relatively well understood in goal-oriented tasks, where automated approaches can be coded by measuring task completion (Hastie, 2012; El Asri et al., 2017; Henderson et al., 2014; Wen et al., 2017; Bordes et al., 2017).”
- MultiWOZ - A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling (2018, methodology): “Subsequently, this approach is followed by Asri et al. (2017) to collect the Frame corpus in a more complex travel booking domain, and Eric et al. (2017) to collect a corpus of conversations for in-car navigation.”
- Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset (2019, methodology): “0 (Wen et al. 2017), FRAMES (El Asri et al. 2017), M2M (Shah et al. 2018) and MultiWOZ (Budzianowski et al. 2018).”
- Key-Value Retrieval Networks for Task-Oriented Dialogue (2017, background): “More recently, Maluuba has released a dataset of hotel and travel-booking dialogues collected in a Wizard-of-Oz Scheme with elaborate semantic frames annotated (Asri et al., 2017).”
- Relevance of Unsupervised Metrics in Task-Oriented Dialogue for Evaluating Natural Language Generation (2017, methodology): “The task-oriented dialogue community should move towards using larger and more complex datasets, which have been recently announced, such as the Frames dataset (El Asri et al., 2017) or the E2E NLG Challenge dataset (Novikova et al., 2016).”
- Hello, It’s GPT-2 - How Can I Help You? Towards the Use of Pretrained Language Models for Task-Oriented Dialogue Systems (2019, background): “MultiWOZ consists of 7 domains and 10 , 438 dialogues and it is substantially larger than previous available datasets (Wen et al., 2017; El Asri et al., 2017).”
- A Dataset for Document Grounded Conversations (2018, methodology): “The Frames dataset (Asri et al., 2017) was collected to solve the problem of frame tracking.”
- Bootstrapping a Neural Conversational Agent with Dialogue Self-Play, Crowdsourcing and On-Line Reinforcement Learning (2018, background): “The Wizard-of-Oz setup (Kelley (1984); Dahlb¨ack et al. (1993)) is a popular approach to collect and annotate task-oriented dialogues via crowd-sourcing for training neural conversational models (Wen et al. (2016); Asri et al. (2017)).”
- Look Before you Speak: Visually Contextualized Utterances (2020, background): “Most traditional conversational datasets [6, 10, 31, 71] are solely text based, and notoriously difficult to collect, relying on narrowly constructed ontologies [31, 55] and highly specific domains [6, 10].”
- Hierarchical Imitation and Reinforcement Learning (2018, background): “For instance, a chatbot assisting a user with reservation and booking for flights and hotels (Peng et al., 2017; El Asri et al., 2017) needs to navigate through multiple turns of conversation.”
- How to Motivate Your Dragon: Teaching Goal-Driven Agents to Speak and Act in Fantasy Worlds (2020, background): “This form of dialogue has traditionally been closely related to specific tasks useful in the context of personal assistants with dialogue interfaces (Henderson et al., 2014; El Asri et al., 2017).”
- Situated Dialogue Learning through Procedural Environment Generation (2021, background): “Goal oriented dialogue has also traditionally been linked to personal assistants with dialogue interfaces (Henderson et al., 2014; El Asri et al., 2017).”
- Why Build an Assistant in Minecraft? (2019, background): “There is a large body of work on goal-oriented dialogue agents which typically focus on one task at a time, for example restaurant booking [93, 35, 14], airline booking [24, 91], or movie recommendation [50].”
- Multi-Level Memory for Task Oriented Dialogs (2018, unknown): “Our experiments on three publicly available datasets ( CamRest676 (Su et al., 2016), InCar assistant (Eric and Man-ning, 2017), Maluuba Frames (El Asri et al., 2017)) show a substantial improvement of 15-25 % in both entity F1 scores, and BLEU scores as compared to existing state of the art…”
- SOLOIST: Few-shot Task-Oriented Dialog with A Single Pre-trained Auto-regressive Model (2020, methodology): “We pre-train the grounded response generation model on dialog corpora (Kim et al., 2019; Rastogi et al., 2019; Byrne et al., 2019; Eric and Manning, 2017; Mrkˇsi´c et al., 2016; Asri et al., 2017) 8 as shown in Table 1.”
- Efficient Intent Detection with Dual Sentence Encoders (2020, background): “…conversation in order to solve a particular task with well-defined semantics, such as booking restau-rants, hotels and flights (Hemphill et al., 1990; Williams, 2012; El Asri et al., 2017), providing tourist information (Budzianowski et al., 2018), or automating customer support (Xu et al., 2017).”
- Recent advances in deep learning based dialogue systems: a systematic survey (2021, background): “Task-oriented Dialogue Asri et al. (2017) JDDC A Chinese customer service dataset,consisting of context-response pairs.”