Out-of-Domain Utterance Detection Using Classification Confidences of Multiple Topics

By Ian Lane, Tatsuya Kawahara, Tomoko Matsui, Satoshi Nakamura. 2007

63 citations recorded by the source corpus.

One significant problem for spoken language systems is how to cope with users' out-of-domain (OOD) utterances which cannot be handled by the back-end application system. In this paper, we propose a novel OOD detection framework, which makes use of the classification confidence scores of multiple topics and applies a linear discriminant model to perform in-domain verification. The verification model is trained using a combination of deleted interpolation of the in-domain data and minimum-classification-error training, and does not require actual OOD data during the training process, thus realizing high portability. When applied to the "phrasebook" system, a single utterance read-style speech task, the proposed approach achieves an absolute reduction in OOD detection errors of up to 8.1 points (40% relative) compared to a baseline method based on the maximum topic classification score. Furthermore, the proposed approach realizes comparable performance to an equivalent system trained on both in-domain and OOD data, while requiring no OOD data during training. We also apply this framework to the "machine-aided-dialogue" corpus, a spontaneous dialogue speech task, and extend the framework in two manners. First, we introduce topic clustering which enables reliable topic confidence scores to be generated even for indistinct utterances, and second, we implement methods to effectively incorporate dialogue context. Integration of these two methods into the proposed framework significantly improves OOD detection performance, achieving a further reduction in equal error rate (EER) of 7.9 points

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding (2021, background): “, 2012) or out-of-domain intent detection (Lane et al., 2006), where the rejection label is not presented during training but all new classes during inference (not shown in training) should be covered by the rejection label.”
Detecting out-of-domain utterances addressed to a virtual personal assistant (2014, methodology): “The utterances which do not belong to any of the covered domains can simply be classified as “out-of-domain” using the classification confidence scores of virtual personal assistant (VPA) domain models [8].”
Out-of-domain Detection based on Generative Adversarial Network (2018, methodology): “2 Related Work Lane et al. (2007) proposed an in–domain veriﬁcation method.”
Out-of-Domain Detection Method Based on Sentence Distance for Dialogue Systems (2018, methodology): “[4] measure the confidence score and perform an ID verification using the bagof-words model.”
Neural sentence embedding using only in-domain sentences for out-of-domain sentence detection in dialog systems (2017, methodology): “[12] proposed an in-domain verification (IDV) method, which uses only ID sentences to build domain-wise one-vs.”
Joint Learning of Domain Classification and Out-of-Domain Detection with Dynamic Class Weighting for Satisficing False Acceptance Rates (2018, methodology): “Also, the methods are evaluated either based on IND recall and OOD recall [5] or Equal Error Rate (ERR), where thresholding is used to match FAR and FRR to be the same [4, 6].”