Use of kernel deep convex networks and end-to-end learning for spoken language understanding

By Li Deng, Gökhan Tür, Xiaodong He, Dilek Z. Hakkani-Tür. 2012

125 citations recorded by the source corpus.

We present our recent and ongoing work on applying deep learning techniques to spoken language understanding (SLU) problems. The previously developed deep convex network (DCN) is extended to its kernel version (K-DCN) where the number of hidden units in each DCN layer approaches infinity using the kernel trick. We report experimental results demonstrating dramatic error reduction achieved by the K-DCN over both the Boosting-based baseline and the DCN on a domain classification task of SLU, especially when a highly correlated set of features extracted from search query click logs are used. Not only can DCN and K-DCN be used as a domain or intent classifier for SLU, they can also be used as local, discriminative feature extractors for the slot filling task of SLU. The interface of K-DCN to slot filling systems via the softmax function is presented. Finally, we outline an end-to-end learning strategy for training the softmax parameters (and potentially all DCN and K-DCN parameters) where the learning objective can take any performance measure (e.g. the F-measure) for the full SLU system.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Deep Learning: Methods and Applications (2014, background): “, 2012), a new deep learning architecture, sometimes called Deep Stacking Network (DSN), together with its tensor variant (Hutchinson et al, 2012, 2013) and its kernel version (Deng et al., 2012), are developed that all focus on discrimination with scalable, parallelizable learning relying on little”
A Survey on Dialogue Systems: Recent Advances and New Frontiers (2017, methodology): “ntent detection is performed to detect the intent of a user. It classiﬁes the utterance into one of the pre-deﬁned intents. Deep learning techniques have been successively applied in intent detection [15; 84; 112]. In particular, [25] used convolutional neural networks (CNN) to extract query vector ”
A tutorial survey of architectures, algorithms, and applications for deep learning (2014, background): “In themost recentwork of [108–110], a newdeep learning architecture, sometimes calledDSN, togetherwith its tensor variant [111, 112] and its kernel version [11], are developed that all focus on discrimination with scalable, parallelizable learning relying on little or no generative component.”
Domain Adaptive Dialog Generation via Meta Learning (2019, methodology): “As illustrated in Algorithm 1, sequicity model is used to combine nat-ural language understanding (NLU), dialog managing and response generation in a seq2seq fashion, while meta-learning is a method to adjust loss function value for better optimization. α and β in the algorithm are the learning rate”
Recent advances in deep learning based dialogue systems: a systematic survey (2021, background): “Deoras and Sarikaya (2013) built a DBN-based sequence tagger. In addition to the NER input features used in traditional taggers, they also combined part of speech (POS) and syntactic features as a part of the input. The recurrent architectures benefited the sequence tagging task in that they could k”
Multi-Domain Joint Semantic Frame Parsing Using Bi-Directional RNN-LSTM (2016, methodology): “With the advances on deep learning, deep belief networks (DBNs) with deep neural networks (DNNs) have first been employed for intent determination in call centers [15], and later for domain classification in personal assistants [16, 17, 18].”
Recurrent neural networks for language understanding (2013, result): “The result of [37] is the best previously published result with lexical plus Wikipedia [38] features, with a slot filling F1 score of 91.”
Random features for Kernel Deep Convex Network (2013, methodology): “Instead of using a large number of hidden units in DCN, Kernel Deep Convex Network (K-DCN) was further proposed to use a kernel trick so that the number of hidden units in each DCN layer approaches infinity [5].”
Using Recurrent Neural Networks for Slot Filling in Spoken Language Understanding (2015, methodology): “[30] and intent determination [13], [39], [50], we focus on”
How to Scale Up Kernel Methods to Be As Good As Deep Neural Nets (2014, methodology): “In the context of automatic speech recognition (ASR), which this paper focuses on, examples of directly using kernel methods were reported [17, 9].”
Ensemble deep learning: A review (2021, methodology): “The kernelized version of DCN, known as kernel deep convex networks (K-DCN), was given in [81], here the number of hidden layer approach infinity via kernel trick.”
A deep convolutional neural network using heterogeneous pooling for trading acoustic invariance with phonetic confusion (2013, background): “Some related DNN architectures have also demonstrated effectiveness in speech understanding and (small scale) image recognition tasks [7][9][35].”
Predicting speech recognition confidence using deep learning with word identity and score features (2013, background): “Kernel Deep Convex Network (K-DCN) was proposed in [10, 11].”
Recent advances in deep learning for speech research at Microsoft (2013, methodology): “We have also explored the use of various versions of deep learning models for this task, with highly promising results reported in [15][50].”
Investigation of recurrent-neural-network architectures and learning methods for spoken language understanding (2013, methodology): “Subsequently these techniques have been applied to intent determination or semantic utterance classification tasks of SLU [24][7].”
Advances and Challenges in Conversational Recommender Systems: A Survey (2021, background): “Answering Yes/No for an attributes Yes [100] Providing an utterance Yes [28, 119] No [266, 112] Providing preferred attribute values Yes [230] No [141]”
A comprehensive review on ensemble deep learning: Opportunities and challenges (2023, methodology): “In Deng et al. (2012) suggested a stacking method based on deep convex network (DCN) to spoken language understanding (SLU) problems.”