Transfer Learning for Speech Recognition on a Budget

By Julius Kunze, Louis Kirsch, Ilia Kurenkov, Andreas Krug, Jens Johannsmeier, Sebastian Stober. 2017

147 citations recorded by the source corpus.

End-to-end training of automated speech recognition (ASR) systems requires massive data and compute resources. We explore transfer learning based on model adaptation as an approach for training ASR models under constrained GPU memory, throughput and training data. We conduct several systematic experiments adapting a Wav2Letter convolutional neural network originally trained for English ASR to the German language. We show that this technique allows faster training on consumer-grade resources while requiring less training data in order to achieve the same accuracy, thereby lowering the cost of training ASR models in other languages. Model introspection revealed that small adaptations to the network’s weights were sufficient for good performance, especially for inner layers.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Speech & Audio

In Signal Processing

Citations in the corpus

Rethinking Evaluation in ASR: Are Our Models Robust Enough? (2020, background): “For another kind of transfer in [4], the authors transferred LibriSpeech trained wav2letter [5] models to German by fine-tuning them on German, with better performance than training from scratch.”
wav2vec: Unsupervised Pre-training for Speech Recognition (2019, neutral): “, 2018) as well as transferring ASR representations from one language to another (Kunze et al., 2017).”
UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data (2021, background): “Transfer learning (Kunze et al., 2017; Huang et al., 2014; Joshi et al., 2020) first trains the model on the high-resource setting and then fine-tune it on the target data-scarce setting to fit its specificities.”
Speech Recognition (2018, unknown): “…pre-training has focused on emotion recogniton (Lian et al., 2018), speaker identification (Ravanelli & Bengio, 2018), phoneme discrimination (Synnaeve & Dupoux, 2016a; van den Oord et al., 2018) as well as transferring ASR representations from one language to another (Kunze et al., 2017).”
OPT: Omni-Perception Pre-Trainer for Cross-Modal Understanding and Generation (2021, background): “In audio and speech processing, pre-training has focused on emotion recognition [24], speaker identification [32], phoneme discrimination [39, 27], transferring ASR representations from one language to another [17], unsupervised representations learning for speech [35], audio representation learning”
Freezing Subnetworks to Analyze Domain Adaptation in Neural Machine Translation (2018, background): “…have been shown with as few as dozens of in-domain training sentences (Miceli Barone et al., 2017), and recent work has explored continued training on single sentences (Farajian et al., 2017; Kothur et al., 2018 cross-lingual transfer learning approaches (Gr´ezl et al., 2014; Kunze et al., 2017).”
Latent Backdoor Attacks on Deep Neural Networks (2019, background): “In a deep learning context, transfer learning has been shown to be effective in vision [5, 11, 42, 43], speech [13, 20, 24, 51], and text [22, 34].”
With Great Training Comes Great Vulnerability: Practical Attacks against Transfer Learning (2018, background): “In a deep learning context, transfer learning has been shown to be effective in vision [18, 52, 51, 15], speech [34, 63, 30, 20], and text [33, 40].”