Large-Scale Domain Adaptation via Teacher-Student Learning

By Jinyu Li, M. Seltzer, Rui Zhao, Yifan Gong. 2017

142 citations recorded by the source corpus.

High accuracy speech recognition requires a large amount of transcribed data for supervised training. In the absence of such data, domain adaptation of a well-trained acoustic model can be performed, but even here, high accuracy usually requires significant labeled data from the target domain. In this work, we propose an approach to domain adaptation that does not require transcriptions but instead uses a corpus of unlabeled parallel data, consisting of pairs of samples from the source domain of the well-trained model and the desired target domain. To perform adaptation, we employ teacher/student (T/S) learning, in which the posterior probabilities generated by the source-domain model can be used in lieu of labels to train the target-domain model. We evaluate the proposed approach in two scenarios, adapting a clean acoustic model to noisy speech and adapting an adults speech acoustic model to children speech. Significant improvements in accuracy are obtained, with reductions in word error rate of up to 44% over the original source model without the need for transcribed data in the target domain. Moreover, we show that increasing the amount of unlabeled data results in additional model robustness, which is particularly beneficial when using simulated training data in the target-domain.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Optimization & Training, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Optimization & Training

In Signal Processing

Citations in the corpus

FSD50K: An Open Dataset of Human-Labeled Sound Events (2020, background): “Mitigating this potential issue could be a research problem involving, for example, data augmentation [83] or domain adaptation [84] techniques.”
Lessons from Building Acoustic Models with a Million Hours of Speech (2019, background): “Recently, student/teacher training has become popular in the speech community for model compression ([26], [27], [28]).”
Cracking the cocktail party problem by multi-beam deep attractor network (2017, methodology): “In [40], it was shown this T/S learning method is very effective in producing accurate target-domain model.”
Unsupervised adaptation with domain separation networks for robust speech recognition (2017, background): “In [21] we show that the source-domain model can be effectively adapted without any transcription by using teacherstudent (T/S) learning [22], in which the posterior probabil-”
Learn Spelling from Teachers: Transferring Knowledge from Language Models to Sequence-to-Sequence Speech Recognition (2019, methodology): “It has also been used for domain adaptation for acoustic models [14] and language models [15].”
Conditional Teacher-student Learning (2019, methodology): “2The parallel data can be either recorded or simulated as in [3].”
Training Keyword Spotting Models on Non-IID Data with Federated Learning (2020, background): “Given the obstacles to on-device labeling, teacher student training can be used to adapt a model trained on the server (with manually labeled data) to the on-device unlabeled data domain [42, 43, 44].”
Knowledge Distillation for Sequence Model (2018, background): “Hence large amount of unlabeled data can be used to make the student model more similar to the teacher [13].”
Cycle-Consistent Speech Enhancement (2018, background): “As shown in [44], teacher-student (T/S) learning [45] is better for robust model adaptation without transcription.”
Recent progresses in deep learning based acoustic models (2017, methodology): “Recently, the teacher/student (T/S) learning [161] method was proposed to perform adaptation without the use of transcriptions.”
Multi-Microphone Neural Speech Separation for Far-Field Multi-Talker Speech Recognition (2018, methodology): “This model was built by using a teacher-student (TS) adaptation technique [19] as follows.”
Show, Attend and Distill: Knowledge Distillation via Attention-based Feature Matching (2021, unknown): “2017; Tian, Krishnan, and Isola 2019), crossdomain transfer learning (Orbes-Arteainst et al. 2019; Li et al. 2017; Asami et al. 2017), and continual learning (Li and Hoiem 2017; Hou et al.”
Improved training for online end-to-end speech recognition systems (2017, background): “This approach has been shown to be successful in several tasks including acoustic modeling [26], speech enhancement [27], and domain adaptation [28].”