WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing

By Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao and 7 more. 2021

3,371 citations recorded by the source corpus.

Self-supervised learning (SSL) achieves great success in speech recognition, while limited exploration has been attempted for other speech processing tasks. As speech signal contains multi-faceted information including speaker identity, paralinguistics, spoken content, etc., learning universal representations for all speech tasks is challenging. To tackle the problem, we propose a new pre-trained model, WavLM, to solve full-stack downstream speech tasks. WavLM jointly learns masked speech prediction and denoising in pre-training. By this means, WavLM does not only keep the speech content modeling capability by the masked speech prediction, but also improves the potential to non-ASR tasks by the speech denoising. In addition, WavLM employs gated relative position bias for the Transformer structure to better capture the sequence ordering of input speech. We also scale up the training dataset from 60 k hours to 94 k hours. WavLM Large achieves state-of-the-art performance on the SUPERB benchmark, and brings significant improvements for various speech processing tasks on their representative benchmarks.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Wei-Ning Hsu
  2. Jinyu Li
  3. Tao Qin
  4. Dong Yu
  5. Xu Tan
  6. Yoshua Bengio
  7. Tie-Yan Liu
  8. Jianfeng Gao

In Representation & Self-Supervised Learning

  1. Wei-Ning Hsu
  2. Alexei Baevski
  3. Abdel-rahman Mohamed
  4. Michael Auli
  5. James R. Glass
  6. David F. Harwath
  7. Yossi Adi
  8. Hung-yi Lee

In Speech & Audio

  1. Wei-Ning Hsu
  2. Jinyu Li
  3. Yossi Adi
  4. Shinji Watanabe
  5. Abdel-rahman Mohamed
  6. Gabriel Synnaeve
  7. Xu Tan
  8. Lei He

In Signal Processing

  1. Jinyu Li
  2. Wei-Ning Hsu
  3. Shinji Watanabe
  4. Yossi Adi
  5. Dong Yu
  6. Xu Tan
  7. Abdel-rahman Mohamed
  8. James R. Glass

Citations in the corpus