WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
By Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao and 7 more. 2021
3,371 citations recorded by the source corpus.
Self-supervised learning (SSL) achieves great success in speech recognition, while limited exploration has been attempted for other speech processing tasks. As speech signal contains multi-faceted information including speaker identity, paralinguistics, spoken content, etc., learning universal representations for all speech tasks is challenging. To tackle the problem, we propose a new pre-trained model, WavLM, to solve full-stack downstream speech tasks. WavLM jointly learns masked speech prediction and denoising in pre-training. By this means, WavLM does not only keep the speech content modeling capability by the masked speech prediction, but also improves the potential to non-ASR tasks by the speech denoising. In addition, WavLM employs gated relative position bias for the Transformer structure to better capture the sequence ordering of input speech. We also scale up the training dataset from 60 k hours to 94 k hours. WavLM Large achieves state-of-the-art performance on the SUPERB benchmark, and brings significant improvements for various speech processing tasks on their representative benchmarks.
Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Speech & Audio, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Representation & Self-Supervised Learning
- Wei-Ning Hsu
- Alexei Baevski
- Abdel-rahman Mohamed
- Michael Auli
- James R. Glass
- David F. Harwath
- Yossi Adi
- Hung-yi Lee
In Speech & Audio
In Signal Processing
Citations in the corpus
- VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers (2024, methodology): “SIM is used to evaluate the speaker similarity between the original prompt and synthesized speech, leveraging the SOTA speaker verification model, WavLM-TDNN 3 [Chen et al., 2022].”
- BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data (2024, methodology): “Specifically, Further, we propose speaker-disentangled speechcodes that are built on top of a WavLM [39] Self-Supervised Learning (SSL) speech model.”
- ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reordering (2024, methodology): “SPK was assessed using the fine-tuned WavLM-TDNN model 4 (Chen et al., 2022), scoring similarity on a scale of -1 to 1, with values above 0.86 indicate the same speaker identity (This value comes from the release model card page).”
- Audiobox: Unified Audio Generation with Natural Language Prompts (2023, methodology): “WavLM-TDCNN (Chen et al., 2022b) is commonly used for embedding (Wang et al., 2023a; Kharitonov et al., 2023; Le et al., 2023).”
- Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale (2023, methodology): “VALL-E proposed to use WavLM-TDCNN speaker embedding model [Chen et al., 2022], which maps an audio clip to a fixed dimensional vector, to measure voice similarity.”
- StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models (2023, methodology): “One of the key contributions of StyleTTS 2 is the use of large pre-trained speech language models (SLMs) like Wav2Vec 2.0 [9], HuBERT [10], and WavLM [11] as discriminators, in conjunction with a novel differentiable duration modeling approach.”
- Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias (2023, methodology): “And for the cosine speaker similarity, we use the WavLM model [12] finetuned for speaker verification9 to compute the cosine speaker similarity score between the ground-truth speech and synthesized speech.”
- VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translation (2023, neutral): “In terms of building one model for all speech tasks, HuBERT (Hsu et al., 2021) and WavLM (Chen et al., 2022) learn a universal speech representation model using unsupervised masked unit prediction methods, like BERT (Devlin et al., 2018).”
- SoundStorm: Efficient Parallel Audio Generation (2023, positive): “As in Wang et al. (2023) and Kharitonov et al. (2023), we compute embeddings with a publicly available 1 speaker verification system based on WavLM (Chen et al., 2022).”
- NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (2023, methodology): “To ensure high-quality generation, we use a speech scoring model [45] to filter the multiple samples generated by the diffusion model with different starting Gaussian noises zT .”
- Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling (2023, positive): “Specifically, we measure speaker similarity between synthesized target speech and groud-truth target speech or source speech as an automatic speaker verification (ASV) task, where a WavLM [Chen et al., 2022] based ASV model is used to calculate the score.”
- SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing (2021, background): “…and speech communities, due to its strong capability of generalization and efficient usage of large-scale data (Devlin et al., 2019; Liu et al., 2019; Yang et al., 2019; Lewis et al., 2020; Chen et al., 2021c; Baevski et al., 2020; Lakhotia et al., 2021; Kharitonov et al., 2021; Chen et al., 2021a).”
- Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentation (2022, background): “Self-supervised speech encoder pre-training has led to huge performance improvement on a wide range of applications [22] such as ASR [20, 23], S2T [18], speaker identification [24], etc.”
- Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervision (2023, background): “We set n s to 3, as a trade-off between audio quality and computational cost (Appendix B).”
- AudioGen: Textually Guided Audio Generation (2022, background): “Chen et al. (2022) proposed a similar version of HuBERT trained on larger and augmented dataset.”
- data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language (2022, background): “…for computer vision has been a very active area of research with methods contrasting representations of augmentations of the same image, entirely different images (Chen et al., 2020; Grill et al., 2020; Caron et al., 2021; Chen et al., 2021b) as well as online clustering (Caron et al., 2020).”
- i-Code V2: An Autoregressive Generation Framework over Vision, Language, and Speech Data (2023, methodology): “We compare i-Code V2 with the following models: WavLM (Chen et al., 2022) Transformer-based speech encoder that is pretrained on audio data with self-supervised learning wav2vec 2.0 (Baevski et al., 2020) A speech representation model with CNN-Transformer architecture, pretrained with a contrastive ”