On Generative Spoken Language Modeling from Raw Audio

By Kushal Lakhotia, Evgeny Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu Nguyen, Jade Copet, Alexei Baevski, Abdel-rahman Mohamed and 1 more. 2021

483 citations recorded by the source corpus.

Abstract We introduce Generative Spoken Language Modeling, the task of learning the acoustic and linguistic characteristics of a language from raw audio (no text, no labels), and a set of metrics to automatically evaluate the learned representations at acoustic and linguistic levels for both encoding and generation. We set up baseline systems consisting of a discrete speech encoder (returning pseudo-text units), a generative language model (trained on pseudo- text), and a speech decoder (generating a waveform from pseudo-text) all trained without supervision and validate the proposed metrics with human evaluation. Across 3 speech encoders (CPC, wav2vec 2.0, HuBERT), we find that the number of discrete units (50, 100, or 200) matters in a task-dependent and encoder- dependent way, and that some combinations approach text-based systems.1

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Speech & Audio

In Signal Processing

Citations in the corpus

Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (2023, positive): “Our VALL-E with Vocos signiﬁcantly outperforms the speech-to-speech base-line GSLM and the zero-shot TTS baseline YourTTS, despite the Authorized licensed use limited to the terms of the applicable license agreement with IEEE.”
High Fidelity Neural Audio Compression (2022, background): “Similarly, Lakhotia et al. (2021); Kharitonov et al. (2021) demonstrated that self-supervised learning methods for speech (e.g., HuBERT (Hsu et al., 2021)), can be quantized and used for conditional and unconditional speech generation.”
Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale (2023, methodology): “The GSLM-family [36, 28, 41] are textless language models built upon HuBERT units [22] for speech continuation without using text.”
AudioPaLM: A Large Language Model That Can Speak and Listen (2023, neutral): “Recent work [Lakhotia et al., 2021, Wang et al., 2023] has explored generating speech by modeling discretized representations as target tokens of an autoregressive Transformer [Vaswani et al., 2017] network.”
UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocoding (2023, methodology): “Textless NLP (Lakhotia et al. 2021) and AudioLM (Borsos et al. 2022) propose to leverage wav2vec 2.0 and w2v-BERT (Chung et al. 2021) respectively for language model training and consequently are able to generate speech unconditionally via auto-regressive inference.”
SoundStorm: Efficient Parallel Audio Generation (2023, neutral): “…et al., 2020; Hsu et al., 2021; Chung et al., 2021) have provided a low-framerate representation of the underlying signal which remains rich enough after discretization for language models to generate intelligible speech from a speciﬁc speaker as a sequence of tokens (Lakhotia et al., 2021).”
AudioLM: A Language Modeling Approach to Audio Generation (2022, methodology): “same set of samples using the unit-to-speech synthesis module of GSLM [14] as provided by textless-lib [46].”
SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing (2021, background): “, 2021) have drawn much attention in the NLP and speech communities, due to its strong capability of generalization and efficient usage of largescale data (Devlin et al., 2019; Liu et al., 2019; Yang et al., 2019; Lewis et al., 2020; Chen et al., 2021c; Baevski et al., 2020; Lakhotia et al., 2021; K”
Speech Resynthesis from Discrete Disentangled Self-Supervised Representations (2021, methodology): “Implementation Details We follow the same setup as in [5].”
Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentation (2022, background): “Most recently, [4] proposes to apply a self-supervised speech encoder pre-trained on unlabeled speech to convert target speech into discrete units [9] and build a speech-to-unit translation (S2UT) model for direct S2ST.”
It's Raw! Audio Generation with State-Space Models (2022, background): “( Right ) Pooling layers are position-wise linear transformations with shifts to ensure causality. architecture [39]—remains a central component of state-of-the-art approaches for text-to-speech (TTS) [26], unconditional generation [25], and non-autoregressive (non-AR) generation [23].”
Textless Speech-to-Speech Translation on Real Data (2021, positive): “(2021) have proposed to take advantage of self-supervised discrete representations (Lakhotia et al., 2021), or discrete units, learned from unlabeled speech data as the target for building a direct S2ST model.”
Text-Free Prosody-Aware Generative Spoken Language Modeling (2021, methodology): “2 The row 1 corresponds to the prosody-ignorant baseline model of (Lakhotia et al., 2021).”
Direct Speech-to-Speech Translation With Discrete Units (2021, methodology): “We use the pre-trained HuBERT model1 trained on Librispeech [28] for two iterations and follow [7, 10] to perform k-means with K = 100 on representations from the sixth layer of the model for extracting discrete units for all target English speech.”
Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervision (2023, methodology): “At its core, SPEAR-TTS leverages recent advances in the “textless” modeling of spoken language (Lakhotia et al., 2021; Dunbar et al., 2021; Polyak et al., 2021; Kreuk et al., 2021; Kharitonov et al., 2022; Nguyen et al., 2022; Borsos et al., 2022).”
AudioGen: Textually Guided Audio Generation (2022, methodology): “Similarly to (Lakhotia et al., 2021), we ﬁrst encode an audio prompt into a sequence of audio tokens, feed it into the ALM, and sample a continuation.”
Are Discrete Units Necessary for Spoken Language Modeling? (2022, background): “In the same spirit, [15] introduced Generative Spoken Language Modeling (GSLM), the task of learning and generating spoken language”
General-purpose, long-context autoregressive modeling with Perceiver AR (2022, methodology): “Similar techniques have been developed for audio, where vector-quantized encodings of raw waveforms have effectively been used as vocabularies for end-to-end speech synthesis (Lakhotia et al., 2021; Polyak et al., 2021) and music generation (Dhariwal et al., 2020).”