BERT for Joint Intent Classification and Slot Filling

By Qian Chen, Zhu Zhuo, Wen Wang. arXiv.org, 2019

629 citations recorded by the source corpus.

Intent classification and slot filling are two essential tasks for natural language understanding. They often suffer from small-scale human-labeled training data, resulting in poor generalization capability, especially for rare words. Recently a new language representation model, BERT (Bidirectional Encoder Representations from Transformers), facilitates pre-training deep bidirectional representations on large-scale unlabeled corpora, and has created state-of-the-art models for a wide variety of natural language processing tasks after simple fine-tuning. However, there has not been much effort on exploring BERT for natural language understanding. In this work, we propose a joint intent classification and slot filling model based on BERT. Experimental results demonstrate that our proposed model achieves significant improvement on intent classification accuracy, slot filling F1, and sentence-level semantic frame accuracy on several public benchmark datasets, compared to the attention-based recurrent neural network models and slot-gated models.

View Paper: https://www.semanticscholar.org/paper/476029ac9be26bf7f121a388f5c1e45d204efe52

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Semi-Supervised Spoken Language Understanding via Self-Supervised Speech and Language Model Pretraining (2020, methodology): “Our framework improves upon previous work in Word Error Rate (WER) and IC/SL on ATIS, and even rivaled its NLU counterpart with oracle text input [10].”
FLIN: A Flexible Natural Language Interface for Web Navigation (2020, unknown): “Models that jointly perform intent detection and slot filling (Guo et al., 2014; Liu and Lane, 2016; Chen et al., 2019) are not applicable to our problem for three reasons.”
A Stack-Propagation Framework with Token-Level Intent Detection for Spoken Language Understanding (2019, methodology): “In addition, our model + BERT outperforms the BERT SLU (Chen et al., 2019) joint the two tasks and there is no explicit interaction between intent detection and slot ﬁlling in two datasets in overall acc metric.”
CINS: Comprehensive Instruction for Few-shot Learning in Task-oriented Dialog Systems (2021, background): “Large-scale pre-trained language models (PLMs), such as BERT (Devlin et al. 2019), UniLM (Dong et al. 2019), GPT2 (Radford et al. 2019), T5 (Raffel et al. 2020) and GPT-3 (Brown et al. 2020), have shown tremendous success in various NLP applications, especially in few-shot or zero-shot learning scen”
DialoGLUE: A Natural Language Understanding Benchmark for Task-Oriented Dialogue (2020, background): “…on large natural language corpora, these models gain generalized language understanding capabilities that transfer effectively to downstream tasks (Wang et al., 2018), including intent prediction (Chen et al., 2019; Castellucci et al., 2019) and dialogue state tracking (Heck et al., 2020).”
An Application-Independent Approach to Building Task-Oriented Chatbots with Interactive Continual Learning (2020, background): “(1) BERT-JISF: This is a state-of-the-art joint intent detection and slot ﬁlling model that ﬁnite-tunes a pre-trained BERT model 3 to solve NLU [8].”
Speech-Language Pre-Training for End-to-End Spoken Language Understanding (2021, methodology): “We use the same data division as [6] and [26, 16] for FSC and ATIS datasets, respectively.”