Self-Supervised Speech Representation Learning: A Review

By Abdel-rahman Mohamed, Hung-yi Lee, Lasse Borgholt, Jakob Drachmann Havtorn, Joakim Edin, Christian Igel, Katrin Kirchhoff, Shang-Wen Li, Karen Livescu, Lars Maaløe and 2 more. 2022

515 citations recorded by the source corpus.

Although supervised deep learning has revolutionized speech and audio processing, it has necessitated the building of specialist models for individual tasks and application scenarios. It is likewise difficult to apply this to dialects and languages for which only limited labeled data is available. Self-supervised representation learning methods promise a single universal model that would benefit a wide variety of tasks and domains. Such methods have shown success in natural language processing and computer vision domains, achieving new levels of performance while reducing the number of labels required for many downstream scenarios. Speech representation learning is experiencing similar progress in three main categories: generative, contrastive, and predictive methods. Other approaches rely on multi-modal data for pre-training, mixing text or visual data streams with speech. Although self-supervised speech representation is still a nascent research area, it is closely related to acoustic word embedding and learning with zero lexical resources, both of which have seen active research for many years. This review presents approaches for self-supervised speech representation learning and their connection to other research areas. Since many current methods focus solely on automatic speech recognition as a downstream task, we review recent efforts on benchmarking learned representations to extend the application beyond speech recognition.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Chenpeng Du
2. Xie Chen
3. K. Yu
4. Xipeng Qiu
5. Daniel Povey
6. Ziyang Ma
7. Thomas Drugman
8. Yifan Yang

In Signal Processing

1. Chenpeng Du
2. Xie Chen
3. K. Yu
4. Daniel Povey
5. Ziyang Ma
6. Thomas Drugman
7. Yifan Yang
8. Dong Zhang

Citations in the corpus

• BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data (2024, background): “While John, who had been working late every night for a month on his novel, finally took a break to enjoy the fresh air, his neighbor, a painter who often found inspiration in the midnight moon, was just beginning her creative process.”
• SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models (2023, methodology): “We employ HuBERT (Hsu et al., 2021) as our semantic teacher in this study, as HuBERT is demonstrated to encompass substantial content information (Mohamed et al., 2022).”
• Towards Universal Speech Discrete Tokens: A Case Study for ASR and TTS (2023, background): “Recalling that the SSL models used for extracting discrete tokens are trained in English corpus, the generalization of current discrete tokens across languages is yet to be improved.”