Self-supervised Learning with Random-projection Quantizer for Speech Recognition

By Chung-Cheng Chiu, James Qin, Yu Zhang, Jiahui Yu, Yonghui Wu. 2022

258 citations recorded by the source corpus.

We present a simple and effective self-supervised learning approach for speech recognition. The approach learns a model to predict the masked speech signals, in the form of discrete labels generated with a random-projection quantizer. In particular the quantizer projects speech inputs with a randomly initialized matrix, and does a nearest-neighbor lookup in a randomly-initialized codebook. Neither the matrix nor the codebook is updated during self-supervised learning. Since the random-projection quantizer is not trained and is separated from the speech recognition model, the design makes the approach flexible and is compatible with universal speech recognition architecture. On LibriSpeech our approach achieves similar word-error-rates as previous work using self-supervised learning with non-streaming models, and provides lower word-error-rates and latency than wav2vec 2.0 and w2v-BERT with streaming models. On multilingual tasks the approach also provides significant improvement over wav2vec 2.0 and w2v-BERT.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Yu Zhang
3. Tara N. Sainath
4. Dong Yu
5. Yonghui Wu
6. Chung-Cheng Chiu
7. Shinji Watanabe
8. Bo Li

In Signal Processing

1. Jinyu Li
2. Tara N. Sainath
3. Yu Zhang
4. Dong Yu
5. Yonghui Wu
6. Chung-Cheng Chiu
7. Shinji Watanabe
8. Bo Li

Citations in the corpus

• SoundStorm: Efficient Parallel Audio Generation (2023, positive): “To extract semantic tokens, we train a 0.6B parameter BEST-RQ (Chiu et al., 2022) on this dataset and fit k -means with 4096 cluster centers to the activations of layer 13 of the BEST-RQ model.”
• BEATs: Audio Pre-Training with Acoustic Tokenizers (2022, methodology): “For the first iteration of BEATS pre-training, we apply the random-projection tokenizer [Chiu et al., 2022] to generate the patch-level discrete labels for each input audio.”
• Textless Direct Speech-to-Speech Translation with Discrete Speech Representation (2022, methodology): “The random quantization is inspired by the BEST-RQ work [25].”
• Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages (2023, methodology): “Various unsupervised pre-training methods for speech models have been proposed and applied in [6, 10, 21].”