Attention-Based Recurrent Neural Network Models for Joint Intent Detection and Slot Filling
721 citations recorded by the source corpus.
Attention-based encoder-decoder neural network models have recently shown promising results in machine translation and speech recognition. In this work, we propose an attention-based neural network model for joint intent detection and slot filling, both of which are critical steps for many speech understanding and dialog systems. Unlike in machine translation and speech recognition, alignment is explicit in slot filling. We explore different strategies in incorporating this alignment information to the encoder-decoder framework. Learning from the attention mechanism in encoder-decoder model, we further propose introducing attention to the alignment-based RNN models. Such attentions provide additional information to the intent classification and slot label prediction. Our independent task models achieve state-of-the-art intent detection error rate and slot filling F1 score on the benchmark ATIS task. Our joint training model further obtains 0.56% absolute (23.8% relative) error reduction on intent detection and 0.23% absolute gain on slot filling over the independent task models.
Subfields: Artificial Intelligence, Natural Language Processing, Information Extraction
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jianfeng Gao
- Bing Liu
- Mohit Bansal
- Luke S. Zettlemoyer
- Richard Socher
- Pascale Fung
- Percy Liang
- Erik Cambria
In Natural Language Processing
- Jianfeng Gao
- Pascale Fung
- Erik Cambria
- Bing Liu
- Andrea Madotto
- Ting-Wei Liu
- Philip S. Yu
- Mohit Bansal
In Information Extraction
Citations in the corpus
- Neural Approaches to Conversational AI (2018, neutral): “data when creating NLU models for a new domain (Hakkani-Tür et al., 2016; Liu and Lane, 2016).”
- Incremental Few-shot Text Classification with Multi-round New Classes: Formulation, Dataset and System (2021, background): “One is intent detection that aims at understanding the intents under user queries (Liu and Lane, 2016; Xia et al., 2018).”
- A Progressive Model to Enable Continual Learning for Semantic Slot Filling (2019, methodology): “…are designed to either train a slot filling model independently (Yao et al., 2014; Peng et al., 2015; Kurata et al., 2016; Hakkani-Tür et al., 2016; Liu and Lane, 2016; Deng et al., 2019; Ray et al., 2019) or jointly with the other intent detection task in SLU (Guo et al., 2014; Liu and Lane,…”
- Towards Open Intent Discovery for Conversational Text (2019, methodology): “[15, 31, 48, 51]) since these are formulated as classification problems, and classify intents into limited, pre-defined categories.”
- HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection (2020, methodology): “BiRNN-Attention This model is identical to the BiRNN model but includes an attention layer (Liu and Lane 2016) 10 The limitation is due to the lack of an attention mechanism.”
- Task-Oriented Dialog Systems that Consider Multiple Appropriate Responses under the Same Context (2019, background): “The trend of building task-oriented systems is changing from training separate dialog models independently (Young et al. 2013; Wen et al. 2015; Liu and Lane 2016; Mrkˇsi´c et al. 2017) to end-to-end trainable dialog models (Zhao et al. 2017; Wen et al. 2017; Eric and Manning 2017).”
- CRUISE: Cold-Start New Skill Development via Iterative Utterance Generation (2018, methodology): “NLU Engines & Performance Metrics: To maximally reduce the bias from NLU engines, we evaluate the performance using different existing NLU engines: open source RASA (RASA, 2017) and deep learning based RNN NLU engine with joint learning of intent classifier and slot tagger (Liu and Lane, 2016).”
- Explainable Artificial Intelligence Applications in Cyber Security: State-of-the-Art in Research (2022, methodology): “Several models including CNN-GRU [172], BiRNN [173], and BiRNNAttention [174] were used and tested on this dataset whereas explainability-based metrics such as IntersectionOver-Union (IOU), comprehensiveness, and sufficiency were utilized to evaluate the model interpretability.”
- Seq2Seq and Multi-Task Learning for joint intent and content extraction for domain specific interpreters (2018, background): “It is believed to be more efficient than the traditional approach of single-task networks as learning linked tasks could help learn each task faster [1].”
- Cross-lingual Transfer Learning for Multilingual Task Oriented Dialog (2018, neutral): “Liu and Lane, 2016).”
- Zero-shot User Intent Detection via Capsule Neural Networks (2018, background): “With recent developments in deep neural networks, user intent detection models (Hu et al., 2009; Xu and Sarikaya, 2013; Zhang et al., 2016; Liu and Lane, 2016; Chen et al., 2016b) are proposed to classify user intents given their diversely”
- Semantic Parsing for Task Oriented Dialog using Hierarchical Representations (2018, neutral): “Most work on task oriented dialog systems has focused on identifying a single user intent and then filling the relevant slots – for example, the representations used on the ATIS dataset (Mesnil et al., 2013; Liu and Lane, 2016; Zhu and Yu, 2017) and in the Dialog State Tracking Challenge (Williams e”
- BERT for Joint Intent Classification and Slot Filling (2019, methodology): “Recently, several joint learning meth-ods for intent classification and slot filling were proposed to exploit and model the dependencies between the two tasks and improve the performance over independent models (Guo et al., 2014; Hakkani-T¨ur et al., 2016; Liu and Lane, 2016; Goo et al., 2018).”
- A Novel Bi-directional Interrelated Model for Joint Intent Detection and Slot Filling (2019, methodology): “Slot filling: The i th slot context vector c is computed as the weighted sum of BLSTM’s hidden states ( h 1 , ..., h t ) : where the attention weight α is acquired the same way as in (Liu and Lane, 2016).”
- Don’t Parse, Generate! A Sequence to Sequence Architecture for Task-Oriented Semantic Parsing (2020, methodology): “90 Attention BiRNN [16] 74.”
- Adversarial Adaptation of Synthetic or Stale Data (2017, background): “…and features: jointly modeling intent and slot predictions (Jeong and Lee, 2008; Xu and Sarikaya, 2013; Guo et al., 2014; Zhang and Wang, 2016; Liu and Lane, 2016a,b), modeling SLU models with web search click logs (Li et al., 2009; Kim et al., 2015a) and enhancing features, including…”