Expanding the Scope of the ATIS Task: The ATIS-3 Corpus

By D. Dahl, M. Bates, Michael Brown, W. Fisher, Kate Hunicke-Smith, D. Pallett, Christine Pao, Alexander I. Rudnicky, Elizabeth Shriberg. 1994

The Air Travel Information System (ATIS) domain serves as the common evaluation task for ARPA spoken language system developers. To support this task, the Multi-Site ATIS Data COllection Working group (MADCOW) coordinates data collection activities. This paper describes recent MADCOW activities. In particular, this paper describes the migration of the ATIS task to a richer relational database and development corpus (ATIS-3) and describes the ATIS-3 corpus. The expanded database, which includes information on 46 US and Canadian cities and 23,457 flights, was released in the fall of 1992, and data collection for the ATIS-3 corpus began shortly thereafter. The ATIS-3 corpus now consists of a total of 8297 released training utterances and 3211 utterances reserved for testing, collected at BBN, CMU, MIT, NIST and SRI. 2906 of the training utterances have been annotated with the correct information from the database. This paper describes the ATIS-3 corpus in detail, including breakdowns of data by type (e.g. context-independent, context-dependent, and unevaluable)and variations in the data collected at different sites. This paper also includes a description of the ATIS-3 database. Finally, we discuss future data collection and evaluation plans.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

QuAC: Question Answering in Context (2018, background): “Such open information seeking was studied in semantic parsing on knowledge bases (Dahl et al., 1994) and more recently with modern approaches (Saha et al.”
SPoC: Search-based Pseudocode to Code (2019, background): “While semantic parsing datasets with shorter, query-like programs [57, 7, 33, 4, 59, 23, 44, 55] usually have an execution environment (e.”
Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task (2018, methodology): “150 man-hours Review 150 Recently, some state-of-the-art methods with Seq2Seq architectures are able to achieve over 80% exact matching accuracy even on some complex benchmarks such as ATIS and GeoQuery.”
Learning End-to-End Goal-Oriented Dialog (2016, background): “3 Goal-Oriented Dialog Tasks All of our tasks involve a restaurant reservation system, where the goal of users in each conversation is to book a table at a restaurant.”
Towards Open Intent Discovery for Conversational Text (2019, methodology): “Moreover, commonly used intent-labeled datasets in dialog research such as SNIPS [6] or ATIS [7, 17] largely have concise, coherent and single-sentence texts.”
Learning Context-Dependent Mappings from Sentences to Logical Form (2009, methodology): “Data In this section, we present experiments in the context-dependent ATIS domain (Dahl et al., 1994).”
NL2Bash: A Corpus and Semantic Parser for Natural Language Interface to the Linux Operating System (2018, background): “1 (Dahl et al., 1994) WebQSP DSL 4,737 – – – – search log (Yih et al.”
Neural Semantic Parsing with Type Constraints for Semi-Structured Tables (2017, methodology): “Early data sets, such as GEOQUERY (Zelle and Mooney, 1996) and ATIS (Dahl et al., 1994), have small domains with only a handful of different predicates.”
Training a Natural Language Generator From Unaligned Data (2015, methodology): “…files for experiments on the BAGEL data set, is available for download on Github.11 In future work, we plan to evaluate our generator on further domains, such as geographic information (Kate et al., 2005), weather reports (Liang et al., 2009), or flight information (Dahl et al., 1994).”
QuaRel: A Dataset and Models for Answering Questions about Qualitative Relationships (2018, background): “…Zettlemoyer and Collins 2005; Berant et al. 2013; Krish-namurthy, Dasigi, and Gardner 2017), using datasets about geography (Zelle and Mooney 1996), travel booking (Dahl et al. 1994), factoid QA over knowledge bases (Berant et al. 2013), Wikipedia tables (Pasupat and Liang 2015), and many more.”
From Eliza to XiaoIce: challenges and opportunities with social chatbots (2018, background): “Since the 1990s, a lot of research has been done on task-completion conversational systems (Price, 1990; Hemphill et al., 1990; Dahl et al., 1994; Walk et al., 2001).”
Towards Ecologically Valid Research on Language User Interfaces (2020, unknown): “Some of them were run as user studies (Grosz, 1974; Kelley, 1984; Dahlbäck et al., 1993; Carbonell, 1983), whereas others aimed to collect data for automatic evaluation purposes (Hemphill et al., 1990; Dahl et al., 1994).”
Question Answering is a Format; When is it Useful? (2019, background): “These scenarios are too varied to enumerate, but a few examples are search queries (Dunn et al., 2017; Kwiatkowski et al., 2019), natural language interfaces to databases (Zelle and Mooney, 1996; Berant et al., 2013; Iyer et al., 2017), and virtual assistants (Dahl et al., 1994).”
Unlocking Compositional Generalization in Pre-trained Models Using Intermediate Representations (2021, background): “Combining RIR with LIR d further achieves signiﬁcant improvements over the baseline, especially for ATIS (from 32.9 to 47.8).”
Graph-to-Tree Neural Networks for Learning Structured Input-Output Translation with Applications to Semantic Parsing and Math Word Problem (2020, methodology): “We evaluate our Graph2Tree on three totally-different benchmark datasets, JOBS, GEO, and ATIS, for the semantic parsing task.”
Benchmarking Meaning Representations in Neural Semantic Parsing (2020, background): “In general, semantic parsing aims to transform a nat-ural language utterance into a logic form, i.e., a formal, machine-interpretable meaning representation (MR) (Zelle and Mooney, 1996; Dahl et al., 1994).”