Knowledge Distillation-Based Representation Learning for Short-Utterance Spoken Language Identification
By Peng Shen, Xugang Lu, Sheng Li, Hisashi Kawai. 2020
21 citations recorded by the source corpus.
With successful applications of deep feature learning algorithms, spoken language identification (LID) on long utterances obtains satisfactory performance. However, the performance on short utterances is drastically degraded even when the LID system is trained using short utterances. The main reason is due to the large variation of the representation on short utterances which results in high model confusion. To narrow the performance gap between long, and short utterances, we proposed a teacher-student representation learning framework based on a knowledge distillation method to improve LID performance on short utterances. In the proposed framework, in addition to training the student model on short utterances with their true labels, the internal representation from the output of a hidden layer of the student model is supervised with the representation corresponding to their longer utterances. By reducing the distance of internal representations between short, and long utterances, the student model can explore robust discriminative representations for short utterances, which is expected to reduce model confusion. We conducted experiments on our in-house LID dataset, and NIST LRE07 dataset, and showed the effectiveness of the proposed methods for short utterance LID tasks.
Subfields: Artificial Intelligence, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yu Zhang
- Bhuvana Ramabhadran
- Andrew Rosenberg
- Zhehuai Chen
- Pedro J. Moreno Mengibar
- Gary Wang
- Tara N. Sainath
- Jinyu Li
In Signal Processing
- Bhuvana Ramabhadran
- Yu Zhang
- Andrew Rosenberg
- Pedro J. Moreno Mengibar
- Zhehuai Chen
- Gary Wang
- Jinyu Li
- Tara N. Sainath
Citations in the corpus
- Injecting Text in Self-Supervised Speech Pretraining (2021, background): “While these representations can be learned with multiple views (objectives), there is no guarantee that the learned representation is optimal for any given task such as ASR, language identification or speaker verification tasks [2, 3, 4].”