Evaluation of Spoken Language Systems: the ATIS Domain

By P. Price. 1990

Progress can be measured and encouraged via standards for comparison and evaluation. Though qualitative assessments can be useful in initial stages, quantifiable measures of systems under the same conditions are essential for comparing results and assessing claims. This paper will address the emerging standards for evaluation of spoken language systems.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Efficient Natural Language Response Suggestion for Smart Reply (2017, background): “Early NLU systems parsed natural language with hand-crafted rules to explicit semantic representations, and used manually written state machines to generate speciﬁc responses from the output of parsing [18].”
Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning (2017, methodology): “The datasets are GeoQuery880 (Tang & Mooney, 2001), ATIS (Price, 1990), Free917 (Cai & Yates, 2013), Overnight (Wang et al., 2015), WebQues-tions (Berant et al., 2013), and WikiTableQues-tions (Pasupat & Liang, 2015).”
Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task (2018, methodology): “150 man-hours Review 150 Recently, some state-of-the-art methods with Seq2Seq architectures are able to achieve over 80% exact matching accuracy even on some complex benchmarks such as ATIS and GeoQuery.”
Compositional Semantic Parsing on Semi-Structured Tables (2015, background): “With 3,929 unique column headers (relations) among 13,396 columns, the tables in the W IKI T ABLE Q UESTIONS dataset contain many more relations than closed-domain datasets such as Geoquery (Zelle and Mooney, 1996) and ATIS (Price, 1990).”
From Eliza to XiaoIce: challenges and opportunities with social chatbots (2018, background): “Since the 1990s, a lot of research has been done on task-completion conversational systems (Price, 1990; Hemphill et al., 1990; Dahl et al., 1994; Walk et al., 2001).”
Speak to your Parser: Interactive Text-to-SQL with Natural Language Feedback (2020, unknown): “Compared to ATIS (Price, Step 1: Find the number of rows of each value of id in browser table.”
Context Dependent Semantic Parsing: A Survey (2020, background): “In ATIS (Price, 1990), crowd-workers raise questions around the detailed scripts describing air travel planning scenarios.”
A Joint Model of Intent Determination and Slot Filling for Spoken Language Understanding (2016, methodology): “ATIS dataset: The ATIS corpus [Price, 1990] is the most commonly used dataset for SLU research.”
Multi-Domain Joint Semantic Frame Parsing Using Bi-Directional RNN-LSTM (2016, methodology): “For investigating the integration of contexts for slot filling, we have experimented with the benchmark ATIS data set [7] for the air travel domain.”
MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages (2022, background): “For the task of NLU, the ATIS dataset has been popular in the NLP community since its first release.”
Prefix-to-SQL: Text-to-SQL Generation from Incomplete User Questions (2021, unknown): “, 2017), ATIS (Price, 1990; Dahl et al., 1994) and Spider (Yu et al.”
Building a Neural Semantic Parser from a Domain Ontology (2018, methodology): “Besides geoquery , commonly used datasets are robocup for coaching advice to soccer agents (Kitano, Tambe, Stone, Veloso, Coradeschi, Osawa, Matsubara, Noda, & Asada, 1997), and atis for air travel information service (Price, 1990).”
Spoken language understanding using long short-term memory neural networks (2014, background): “In this paper we focus on spoken language understanding (SLU), in particular, word labeling with semantic information [16] [17] [18] [19] [20] [21] [22].”
Multi-domain spoken language understanding with transfer learning (2009, background): “Spoken language understanding (SLU) has been extensively studied as a component of spoken dialog systems (SDS) to infer speaker intentions and to provide natural human–computer interfaces (Price, 1990; Peckham, 1991; Walker et al., 2002).”