Structure-Level Knowledge Distillation For Multilingual Sequence Labeling

By Xinyu Wang, Yong Jiang, Nguyen Bach, Tao Wang, Fei Huang, Kewei Tu. 2020

40 citations recorded by the source corpus.

Multilingual sequence labeling is a task of predicting label sequences using a single unified model for multiple languages. Compared with relying on multiple monolingual models, using a multilingual model has the benefit of a smaller model size, easier in online serving, and generalizability to low-resource languages. However, current multilingual models still underperform individual monolingual models significantly due to model capacity limitations. In this paper, we propose to reduce the gap between monolingual models and the unified multilingual model by distilling the structural knowledge of several monolingual models (teachers) to the unified multilingual model (student). We propose two novel KD methods based on structure-level information: (1) approximately minimizes the distance between the student’s and the teachers’ structure-level probability distributions, (2) aggregates the structure-level knowledge to local distributions and minimizes the distance between two local probability distributions. Our experiments on 4 multilingual tasks with 25 datasets show that our approaches outperform several strong baselines and have stronger zero-shot generalizability than both the baseline model and teacher models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Fei Huang
2. Nguyen Bach
3. Ricardo Henao
4. Tong Yu
5. Sungchul Kim
6. Yong Jiang
7. Rui Wang
8. Kewei Tu

In Computer Vision and Pattern Recognition

1. Fei Huang
2. Ricardo Henao
3. Tong Yu
4. Sungchul Kim
5. Rui Wang
6. Kewei Tu
7. Handong Zhao
8. Chunyuan Li

Citations in the corpus

• Few-Shot Class-Incremental Learning for Named Entity Recognition (2022, methodology): “Following the current state-of-the-art approach of NER distillation (Wang et al., 2020b), we approximate the sequence-level output distribution of CRF with only its top S predictions.”
• Improving Named Entity Recognition by External Context Retrieving and Cooperative Learning (2021, methodology): “In speech recognition (Huang et al., 2018) and natural language processing (Wang et al., 2020a, 2021b), the marginal probability distribution of the linear-chain CRF layer has been applied to distill the knowledge between teacher models and student models.”
• A Primer on Pretrained Multilingual Language Models (2021, unknown): “…et al., 2020a; K et al., 2020; Dufter and Schütze, 2020; Liu et al., 2020a; Lauscher et al., 2020; Liu et al., 2020c; Conneau and Lample, 2019; Wang et al., 2019; Liu et al., 2019a; Cao et al., 2020; Wang et al., 2020d; Zhao et al., 2020; Wang et al., 2020b; Chi et al., 2020b) have studied…”