Transfer learning for speech and language processing

By Dong Wang, Thomas Fang Zheng. 2015

236 citations recorded by the source corpus.

Transfer learning is a vital technique that generalizes models trained for one setting or task to other settings or tasks. For example in speech recognition, an acoustic model trained for one language can be used to recognize speech in another language, with little or no re-training data. Transfer learning is closely related to multi-task learning (cross-lingual vs. multilingual), and is traditionally studied in the name of `model adaptation'. Recent advance in deep learning shows that transfer learning becomes much easier and more effective with high-level abstract features learned by deep models, and the `transfer' can be conducted not only between data distributions and data types, but also between model structures (e.g., shallow nets and deep nets) or even model types (e.g., Bayesian models and neural models). This review paper summarizes some recent prominent research towards this direction, particularly for speech and language processing. We also report some results from our group and highlight the potential of this very interesting research field1.

Subfields: Artificial Intelligence, Natural Language Processing, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Speech & Audio

Citations in the corpus

Refining Language Models with Compositional Explanations (2021, background): “Sequential transfer learning, also known as model transfer [46], considers a model being trained sequentially over different labeled datasets that are not available at the same time.”
Transfer Learning for Low-Resource Neural Machine Translation (2016, methodology): “In natural language processing, transfer learning methods have been successfully applied to speech recognition, document classiﬁcation and sentiment analysis (Wang and Zheng, 2015).”
How Transferable are Neural Networks in NLP Applications? (2016, background): “It should be noticed that in image or speech processing (Yosinski et al., 2014; Wang and Zheng, 2015), the input of neural networks pretty much consists of raw signals; hence, low-level feature detectors are almost always transferable, even if Yosinski et al.”
Towards Making the Most of BERT in Neural Machine Translation (2019, background): “…embedding (Mikolov et al., 2013; Pennington et al., 2014) to improve performance on supervised tasks, such as named entity recognition, POS tagging semantic role labelling, clas-siﬁcation, and sentiment analysis (Collobert et al., 2011; Socher et al., 2013; Wang and Zheng, 2015; Tan et al., 2018).”
A Neural Grammatical Error Correction System Built On Better Pre-training and Sequential Transfer Learning (2019, background): “This model transfer approach (Wang and Zheng, 2015) can be viewed as a (relatively simple) version of sequential transfer learning (Ruder, 2019).”
Investigation of transfer learning for ASR using LF-MMI trained neural networks (2017, methodology): “al [12] gives a good overall survey of methods used in speech processing.”
Latent Backdoor Attacks on Deep Neural Networks (2019, background): “In a deep learning context, transfer learning has been shown to be effective in vision [5, 11, 42, 43], speech [13, 20, 24, 51], and text [22, 34].”
Speech Model Pre-training for End-to-End Spoken Language Understanding (2019, background): “The computer vision [15, 16], natural language processing [17–21], and ASR [22–25] communities have attacked the problem of limited supervised training data with great success by pre-training deep models on related tasks for which there is more training data.”
With Great Training Comes Great Vulnerability: Practical Attacks against Transfer Learning (2018, background): “In a deep learning context, transfer learning has been shown to be effective in vision [18, 52, 51, 15], speech [34, 63, 30, 20], and text [33, 40].”
Transfer learning for biomedical named entity recognition with neural networks (2018, methodology): “The idea has been successfully applied to many fields, such as speech recognition (Wang and Zheng, 2015), finance (Stamate et al.”