Unsupervised Speech Recognition

By Alexei Baevski, Wei-Ning Hsu, Alexis Conneau, Michael Auli. 2021

302 citations recorded by the source corpus.

Despite rapid progress in the recent past, current speech recognition systems still require labeled training data which limits this technology to a small fraction of the languages spoken around the globe. This paper describes wav2vec-U, short for wav2vec Unsupervised, a method to train speech recognition models without any labeled data. We leverage self-supervised speech representations to segment unlabeled audio and learn a mapping from these representations to phonemes via adversarial training. The right representations are key to the success of our method. Compared to the best previous unsupervised work, wav2vec-U reduces the phoneme error rate on the TIMIT benchmark from 26.1 to 11.3. On the larger English Librispeech benchmark, wav2vec-U achieves a word error rate of 5.9 on test-other, rivaling some of the best published systems trained on 960 hours of labeled data from only two years ago. We also experiment on nine other languages, including low-resource languages such as Kyrgyz, Swahili and Tatar.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Speech & Audio

In Signal Processing

Citations in the corpus

mSLAM: Massively multilingual joint pre-training for speech and text (2022, background): “Pre-trained speech representations have been shown to be close to text (Baevski et al., 2021) and SLAM leverages this similarity for cross-modal transfer.”
On Generative Spoken Language Modeling from Raw Audio (2021, neutral): “…al., 2017; Chung et al., 2019; Hsu et al., 2021), to the extent that Automatic Speech Recognition (ASR) on par with the state of the art from two years back can be built with 5000 times less labelled speech (Baevski et al., 2020b), or even no with no labelled speech at all (Baevski et al., 2021).”
WENETSPEECH: A 10000+ Hours Multi-Domain Mandarin Corpus for Speech Recognition (2021, neutral): “0 [18], Hubert [19], and wav2vec-U [20], etc.”
data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language (2022, background): “This idea was partly inspired by the fact that the top layers of wav2vec 2.0 do not perform as well for downstream tasks as layers in the middle of the network (Baevski et al., 2021; Pasad et al., 2021).”
Scaling Speech Technology to 1, 000+ Languages (2023, methodology): “The most prominent one is wav2vec 2.0 [Baevski et al., 2020b] which enables building speech recognition models with only ten minutes of labeled data and even no labeled data at all [Baevski et al., 2021].”
Layer-Wise Analysis of a Self-Supervised Speech Representation Model (2021, background): “Some very recent work has begun to explore the phonetic content in pre-trained models using a classifier probe [5] and relationships between models trained with different training objec-”
A Cookbook of Self-Supervised Learning (2023, background): “For example convolutions can be useful [Oord et al., 2016, Schneider et al., 2019, Baevski et al., 2021].”
SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Training (2021, background): “Recent work has also shown that self-supervised speech representations can be aligned to text with little to no supervision (Baevski et al., 2021), suggesting the possibility of learning both modalities within a single neural network.”
Self-Supervised Speech Representation Learning: A Review (2022, background): “Wav2vec-U [268] achieved state-of-the-art results at the time,”
SpeechLM: Enhanced Speech Pre-Training With Unpaired Textual Data (2022, background): “Following[33],werandomlyinsertthesilencephoneme ( < SIL > ) between words with a probability of 25%.”
A Brief Overview of Unsupervised Neural Speech Representation Learning (2022, background): “This makes self-supervised learning tie naturally with semi-supervised learning, but it may also be part of a fully unsupervised setup (Baevski et al. 2021).”
XTREME-S: Evaluating Cross-lingual Speech Representations (2022, unknown): “0 [1] have emerged as the predominant approach for automatic speech recognition (ASR) and direct speech-to-text translation (ST), and have made speech models much more data efficient: ASR models can be learnt with as little as a few hours of labeled data [2, 3].”