Knowledge distillation across ensembles of multilingual models for low-resource languages

By Jia Cui, Brian Kingsbury, Bhuvana Ramabhadran, George Saon, Tom Sercu, Kartik Audhkhasi, Abhinav Sethy, Markus Nußbaum-Thom, Andrew Rosenberg. 2017

61 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Natural Language Processing, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Dacheng Tao
3. Wei-Ning Hsu
4. Dong Yu
5. Yifan Gong
6. Gabriel Synnaeve
7. Li Deng
8. Awni Y. Hannun

In Natural Language Processing

1. Awni Y. Hannun
2. Wei-Ning Hsu
3. Gabriel Synnaeve
4. Jinyu Li
5. Ann Lee
6. Zeyuan Allen-Zhu
7. Dacheng Tao
8. John H. L. Hansen

In Efficiency & Compression

1. Jinyu Li
2. Dacheng Tao
3. Yifan Gong
4. Yong Zhao
5. Gakuto Kurata
6. Zhong Meng
7. Jianping Gou
8. Zeyuan Allen-Zhu

Citations in the corpus

• End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures (2019, neutral): “, 2019a) for both low-resource (Veselỳ et al., 2017; Cui et al., 2017) and large-scale (Parthasarathi & Strom, 2019) setups.”
• Knowledge Distillation: A Survey (2020, methodology): “For multilingual representations, knowledge distillation well realized the knowledge transfer among the multi-lingual word embeddings for bilingual dictionary induction (Nakashole and Flauger, 2017), and the effectiveness of the knowledge transfer was also studied cross the ensembles of multilingual”
• Semi-Supervised Speech Recognition via Local Prior Matching (2020, background): “Knowledge distillation (KD) (Cui et al., 2017; Parthasarathi & Strom, 2019) and weak distillation (also known as selftraining or pseudo-labeling) (Veselỳ et al., 2017; Manohar et al., 2018; Li et al., 2019; Kahn et al., 2019a) have also achieved great success in semi-supervised learning for ASR.”
• Efficient Knowledge Distillation from an Ensemble of Teachers (2017, methodology): “This technique has been shown to be very successful in various settings - fully supervised [9], semi-supervised [6], multilingual [11], sequence training [12] - to train student networks perform better than training similar models from scratch using hard targets.”
• Conditional Teacher-student Learning (2019, background): “ASRU, Dec 2011, pp. 24–29.”
• Advancing Multi-Accented Lstm-CTC Speech Recognition Using a Domain Specific Student-Teacher Learning Paradigm (2018, methodology): “This technique has been successfully applied in the ASR domain: distant-talking ASR [17], multilingual [18, 19], domain adaptation [20], and others.”
• Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning (2020, background): “Nor can they explain why the superior performance of ensemble can also be distilled into a single model using knowledge distillation [21, 23, 30, 35, 43, 61, 91].”