vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations

By Alexei Baevski, Steffen Schneider, Michael Auli. 2019

744 citations recorded by the source corpus.

We propose vq-wav2vec to learn discrete representations of audio segments through a wav2vec-style self-supervised context prediction task. The algorithm uses either a gumbel softmax or online k-means clustering to quantize the dense representations. Discretization enables the direct application of algorithms from the NLP community which require discrete inputs. Experiments show that BERT pre-training achieves a new state of the art on TIMIT phoneme classification and WSJ speech recognition.

Subfields: Artificial Intelligence, Natural Language Processing, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Speech & Audio

In Signal Processing

Citations in the corpus

CM3: A Causal Masked Multimodal Model of the Internet (2022, methodology): “The method of tokenizing non-discrete modalities to use standard sequence modeling approaches have been extensively explored with DALL-E for images, Jukebox for Music (Dhariwal et al., 2020) and vq-wav2vec for Speech (Baevski et al., 2019).”
wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations (2020, methodology): “on labeled data with a Connectionist Temporal Classification (CTC) loss [14, 4] to be used for downstream speech recognition tasks (§ 3) Previous work learned a quantization of the data followed by a contextualized representations with a self-attention model [5, 4], whereas our approach solves both ”
XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale (2021, background): “…2.0 contains a convolutional feature encoder f : X 7→ Z to map raw audio X to latent speech representations z 1 , . . . to , z T which are input a Transformer g : Z 7→ C to output context representations c 1 , . . . Training , c T [24]. solves a contrastive task over masked feature encoder outputs.”
Unispeech-Sat: Universal Speech Representation Learning With Speaker Aware Pre-Training (2021, methodology): “Self-supervised learning methods are categorized into discriminative methods [1, 2, 3, 4, 5, 6, 7], generative methods [8, 9, 10, 11, 12, 13], and multi-task learning methods [14].”
HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units (2021, methodology): “configurations, which is composed of seven 512-channel layers with strides [5,2,2,2,2,2,2] and kernel widths [10,3,3,3,3,2,2].”
Unsupervised Speech Recognition (2021, methodology): “Wav2vec-U, or wav2vec Unsupervised, leverages self-supervised representations from wav2vec 2.0 (Baevski et al., 2020c) to embed the speech audio and to segment the ∗.”
Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training (2021, methodology): “The quantization module uses a Gumbel softmax to choose entries from G = 2 codebooks with V = 320 entries each and the chosen entries are concatenated to obtain q [24, 25, 20].”
Multitask Training with Text Data for End-to-End Speech Recognition (2020, background): “Recently, substantial performance improvements have been made by leveraging audio-only data for speech recognition [14, 15, 16].”
Self-Training and Pre-Training are Complementary for Speech Recognition (2020, positive): “This model contains a convolutional feature encoder f : X (cid:55)→ Z to map raw audio X to latent speech representations z 1 , . . . , z T which are input to a Transformer g : Z (cid:55)→ C to output context representations c 1 , . . . , c T [26, 18, 27].”
Learning Robust and Multilingual Speech Representations (2020, neutral): “…representation learning algorithms have shown signiﬁcant improvements at learning representations that correlate well with phonetic structure (van den Oord et al., 2018; Kahn et al., 2019b) and improving downstream speech recognition performance (Schneider et al., 2019; Baevski et al., 2019).”
Unsupervised Pre-Training of Bidirectional Speech Encoders via Masked Reconstruction (2020, methodology): “Besides the concurrent work of [22], we believe our work is also the first to use BERT-style masked reconstruction for representation learning for speech recognition.”
Libri-Light: A Benchmark for ASR with Limited or No Supervision (2019, neutral): “Yet another approach might apply language modeling techniques directly on unlabelled audio to improve the representations before ﬁne-tuning them [26, 27].”
Effectiveness of self-supervised pre-training for speech recognition (2019, methodology): “…model (Devlin et al., 2018; Liu et al., 2019) with only the masked language modeling task on each set of inputs in the same way as described in (Baevski et al., 2019b), namely by choosing tokens for masking with probability of 0.05, expanding each chosen token to a span of 10 masked tokens…”
Towards Unsupervised Speech Recognition and Synthesis with Quantized Speech Representation Learning (2019, background): “Some recent works [8, 9, 10] successfully performed phonetic clustering to a good degree with proper quantization during learning the representations.”
Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (2023, positive): “Recently, vector quantization is widely applied in self-supervised speech models for feature extraction, such as vq-wav2vec [76] and HuBERT [30].”
UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocoding (2023, methodology): “Semantic tokens, such as vq-wav2vec (Baevski, Schneider, and Auli 2019), wav2vec 2.0 (Baevski et al. 2020) and HuBERT (Hsu et al. 2021), are trained for discrimination or masking prediction.”
AudioLM: A Language Modeling Approach to Audio Generation (2022, methodology): “quantizing representations optimized for future time step prediction [32], starting from quantizing low-level audio features followed by iterations of quantization target refinement [33], and jointly learning the quantization along with the masked language model [17].”