An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction
By Stefan Larson, Anish Mahendran, Joseph Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, M. Laurenzano, Lingjia Tang and 1 more. 2019
648 citations recorded by the source corpus.
Task-oriented dialog systems need to know when a query falls outside their range of supported intents, but current text classification corpora only define label sets that cover every example. We introduce a new dataset that includes queries that are out-of-scope—i.e., queries that do not fall into any of the system’s supported intents. This poses a new challenge because models cannot assume that every query at inference time belongs to a system-supported intent class. Our dataset also covers 150 intent classes over 10 domains, capturing the breadth that a production task-oriented agent must handle. We evaluate a range of benchmark classifiers on our dataset along with several different out-of-scope identification schemes. We find that while the classifiers perform well on in-scope intent classification, they struggle to identify out-of-scope queries. Our dataset and evaluation fill an important gap in the field, offering a way of more rigorously and realistically benchmarking text classification in task-driven dialog systems.
Subfields: Artificial Intelligence, Evaluation & Benchmarks, Dialogue & Conversational AI
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Caiming Xiong
- Richard Socher
- Mohit Bansal
- Jianfeng Gao
- Luke S. Zettlemoyer
- Percy Liang
- Graham Neubig
- Zhiyuan Liu
In Evaluation & Benchmarks
- Caiming Xiong
- Dustin Tran
- Wenhao Liu
- Dilek Z. Hakkani-Tür
- Steven C. H. Hoi
- Stanislav Fort
- Ivan Vulic
- Shreyas Padhy
In Dialogue & Conversational AI
- Dilek Z. Hakkani-Tür
- Philip S. Yu
- Matthew Henderson
- Caiming Xiong
- Shikib Mehri
- Fei Mi
- Andrea Madotto
Citations in the corpus
- Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness (2020, methodology): “Detecting Out-of-Scope Intent in Conversational Language Understanding To validate the method beyond image modalities, we also evaluate SNGP on a practical language understanding task where uncertainty quantification is of natural importance: dialog intent detection [43, 76, 79, 81].”
- Lifelong Intent Detection via Multi-Strategy Rebalancing (2021, methodology): “Based on four widely used datasets, ATIS [10], SNIPS [5], HWU64 [16], CLINC150 [14], we constructed four benchmarks.”
- Prompt Augmented Generative Replay via Supervised Contrastive Learning for Lifelong Intent Detection (2022, background): “Lifelong Intent Detection : CLINC150 (Larson et al., 2019) is a crowdsourced multi-domain ( 10 domains such as utility, travel etc.) intent detection dataset.”
- Neural Data Augmentation via Example Extrapolation (2021, methodology): “For this task, we evaluate on the CLINC150 dataset (Larson et al., 2019).”
- DialoGLUE: A Natural Language Understanding Benchmark for Task-Oriented Dialogue (2020, background): “CLINC150 (Larson et al., 2019) contains 23,700 utterances spanning 10 domains (e.g., travel, kitchen/dining, utility, small talk, etc.) and 150 different intent classes.”
- Discriminative Nearest Neighbor Few-Shot Intent Detection by Transferring Natural Language Inference (2020, methodology): “Actually, the OOS recall scores in Larson et al. (2019) and Wu et al. (2020) are much lower than those with our RoBERTa clas-sifier, and the trade-off with respect to the tuning process was not discussed.”
- CINS: Comprehensive Instruction for Few-shot Learning in Task-oriented Dialog Systems (2021, methodology): “OOS For intent classification, we use a benchmark dataset from Larson et al. (2019). Apart from the single out-ofscope intent, it contains 150 intents in 15 domains.”
- Pretrained Language Model in Continual Learning: A Comparative Study (2022, background): “Comparing the performance on different datasets, we notice that PLMs often achieve better performance on more evenly distributed data (i.e. CLINC150 better than Maven better than WebRED).”
- Efficient Intent Detection with Dual Sentence Encoders (2020, background): “It follows the very recent endeavor of procuring high-quality intent detection data (Liu et al., 2019a; Larson et al., 2019), but is very different in nature than the other datasets.”
- Prompt Conditioned VAE: Enhancing Generative Replay for Lifelong Learning in Task-Oriented Dialogue (2022, positive): “We evaluate the PCLL method on intent detection and slot filling based on public NLU benchmarks: For intent detection, we collect six datasets that carry intent annotations: HWU (Liu et al., 2019), BANKING (Casanueva et al., 2020), CLINC (Lar-son et al., 2019), SNIPS (Coucke et al., 2018), AITS (Hemp”
- Meta-learning for Few-shot Natural Language Processing: A Survey (2020, background): “Existing datasets, such as FewRel and CLINC150 mainly consider homogeneous learning tasks; (ii) We are eventually hoping that the model can generalize to tasks of new distributions, as Bansal et al. (2019) did.”
- Continual Few-shot Intent Detection (2022, background): “Traditionally, the ID system is trained with plenty of labeled data to identify a predefined set of intent classes (Larson et al., 2019).”
- A Simple Fix to Mahalanobis Distance for Improving Near-OOD Detection (2021, background): “CIFAR-100, (iii) Genomics OOD benchmark [20] and (iv) CLINC Intent OOD benchmark [11; 13].”
- TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogue (2020, background): “• OOS (Larson et al., 2019): The out-of-scope intent dataset is one of the largest annotated intent datasets, including 15,100/3,100/5,500 samples for the train, validation, and test sets, respec-”