Feature Representation of Short Utterances Based on Knowledge Distillation for Spoken Language Identification

By Peng Shen, Xugang Lu, Sheng Li, Hisashi Kawai. 2018

24 citations recorded by the source corpus.

The performance of spoken language identification (LID) on short utterances is drastically degraded even though model is completely trained on short utterance data set. The degradation is because of the large pattern confusion caused by the large variation of feature representation on short utterances. In this paper, we propose a teacher-student network learning algorithm to explore discriminative features for short utterances. With the teacher-student network learning, the feature representation for short utterances (explored by the student network) are normalized to their representations corresponding to long utterances (provided by the teacher network). With this learning algorithm, the feature representation on short utterances is supposed to reduce pattern confusion. Experiments on a 10-language LID task were carried out to test the algorithm. Our results showed the proposed algorithm significantly improved the performance.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dacheng Tao
2. Jianping Gou
3. S. Maybank
4. Baosheng Yu
5. Hisashi Kawai
6. Xugang Lu
7. Sheng Li
8. Peng Shen

Citations in the corpus

• Knowledge Distillation: A Survey (2020, methodology): “In particular, these KD-based speech recognition applications have spoken language identification (Shen et al. 2018, 2019c, 2020), audio classification (Gao et al. 2019; Perez et al. 2020), text-independent speaker recognition (Ng et al. 2018), speech enhancement (Watanabe et al. 2017), acoustic…”
• MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models (2023, background): “Knowledge Distillation [21, 24] transfers knowledge from a larger teacher model to a smaller student model, achieving promising performance in many applications [22, 48, 56, 63], Recently, [19, 25– 27, 37, 45, 64] propose to transfer reasoning abilities from LLMs ( e.g. , GPT-3.”
• Interactive Learning of Teacher-student Model for Short Utterance Spoken Language Identification (2019, methodology): “However, the performance of the DCNN model decreases rapidly as the input sentence becomes shorter, even the model is completely trained on short utterances [14].”
• Knowledge Distillation-Based Representation Learning for Short-Utterance Spoken Language Identification (2020, methodology): “Inspired by the previous KD works, we investigated to use an internal representation (outputs of one hidden layer of a longutterance-based teacher model) to supervise a short-utterancebased student model, and thereby extract a robust internal representation in our preliminary study [20].”