A Progressive Model to Enable Continual Learning for Semantic Slot Filling

By Yilin Shen, Xiangyu Zeng, Hongxia Jin. 2019

36 citations recorded by the source corpus.

Semantic slot filling is one of the major tasks in spoken language understanding (SLU). After a slot filling model is trained on precollected data, it is crucial to continually improve the model after deployment to learn users’ new expressions. As the data amount grows, it becomes infeasible to either store such huge data and repeatedly retrain the model on all data or fine tune the model only on new data without forgetting old expressions. In this paper, we introduce a novel progressive slot filling model, ProgModel. ProgModel consists of a novel context gate that transfers previously learned knowledge to a small size expanded component; and meanwhile enables this new component to be fast trained to learn from new data. As such, ProgModel learns the new knowledge by only using new data at each time and meanwhile preserves the previously learned expressions. Our experiments show that ProgModel needs much less training time and smaller model size to outperform various model fine tuning competitors by up to 4.24% and 3.03% on two benchmark datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Bing Liu
2. Zixuan Ke
3. Hu Xu
4. Lei Shu
5. Bin Liu
6. Minlie Huang
7. Gyuhak Kim
8. B. Liu

In Computer Vision and Pattern Recognition

1. Zixuan Ke
2. Bin Liu
3. Bing Liu
4. Hu Xu
5. Gyuhak Kim
6. Lei Shu
7. Nianzu Ma
8. Tatsuya Konishi

In Natural Language Processing

1. Hu Xu
2. Bing Liu
3. Lei Shu
4. Zixuan Ke
5. Xiaoyan Zhu
6. Minlie Huang
7. Nianzu Ma
8. Tatsuya Konishi

In Meta & Continual Learning

1. Bing Liu
2. Zixuan Ke
3. Bin Liu
4. Gyuhak Kim
5. Hu Xu
6. Lei Shu
7. Tatsuya Konishi
8. Nianzu Ma

Citations in the corpus

• Continual Training of Language Models for Few-Shot Learning (2022, methodology): “It has been used for slot filling (Shen et al., 2019), language learning (Li et al., 2019), sentence embedding (Liu et al., 2019a), translation (Khayral-lah et al., 2018), cross-lingual modeling (Liu et al., 2020b), question answering (Greco et al., 2019) and text classification (Ke et al., 2021a,b;…”
• Continual Prompt Tuning for Dialog State Tracking (2022, methodology): “For continual learning in dialog system, variants of general CL methods have been applied (Lee, 2017; Shen et al., 2019; Wu et al., 2019; Mi et al., 2020; Geng et al., 2021).”
• Achieving Forgetting Prevention and Knowledge Transfer in Continual Learning (2021, background): “For example, CL models have been proposed for sentiment analysis [23, 24, 37, 43], dialogue slot filling [53], language modeling [58, 7], language learning [31], sentence embedding [33], machine translation [25], cross-lingual modeling [35], and question answering [12].”
• Continual Learning for Natural Language Generation in Task-oriented Dialog Systems (2020, positive): “Shen et al. (2019) recently applied progressive network (Rusu et al., 2016) for the semantic slot filling task from a continual learning perspective similar to ours.”
• Continual Pre-training of Language Models (2023, methodology): “In NLP, CL has been used for slot filling (Shen et al., 2019), language learning (Li et al., 2019), sentiment analysis (Ke et al., 2021a), topic modeling (Gupta et al., 2020), question answering (Greco et al., 2019) and text classification (Sun et al., 2020; Huang et al., 2021; Chuang et al., 2020).”