Text-Free Prosody-Aware Generative Spoken Language Modeling

By Evgeny Kharitonov, Ann Lee, Adam Polyak, Yossi Adi, Jade Copet, Kushal Lakhotia, Tu Nguyen, Morgane Rivière, Abdel-rahman Mohamed, Emmanuel Dupoux and 1 more. 2021

159 citations recorded by the source corpus.

Speech pre-training has primarily demonstrated efficacy on classification tasks, while its capability of generating novel speech, similar to how GPT-2 can generate coherent paragraphs, has barely been explored. Generative Spoken Language Modeling (GSLM) (CITATION) is the only prior work addressing the generative aspect of speech pre-training, which builds a text-free language model using discovered units. Unfortunately, because the units used in GSLM discard most prosodic information, GSLM fails to leverage prosody for better comprehension and does not generate expressive speech. In this work, we present a prosody-aware generative spoken language model (pGSLM). It is composed of a multi-stream transformer language model (MS-TLM) of speech, represented as discovered unit and prosodic feature streams, and an adapted HiFi-GAN model converting MS-TLM outputs to waveforms. Experimental results show that the pGSLM can utilize prosody to improve both prosody and content modeling, and also generate natural, meaningful, and coherent speech given a spoken prompt. Audio samples can be found at https://speechbot.github.io/pgslm. Codes and models are available at https://github.com/pytorch/fairseq/tree/main/examples/textless_nlp/pgslm.

Subfields: Artificial Intelligence, Generative Models, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Generative Models

In Speech & Audio

Citations in the corpus

High Fidelity Neural Audio Compression (2022, methodology): “Similar methods were applied to speech resynthesis (Polyak et al., 2021), speech emotion conversion (Kreuk et al., 2021), spoken dialog system (Nguyen et al., 2022), and speech-to-speech translation (Lee et al., 2021a;b; Popuri et al., 2022).”
Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale (2023, methodology): “The GSLM-family [36, 28, 41] are textless language models built upon HuBERT units [22] for speech continuation without using text.”
AudioLM: A Language Modeling Approach to Audio Generation (2022, methodology): “We note that our proposal for the extraction of semantic tokens from w2v-BERT resembles the token extraction from HuBERT in prior works [14], [45].”
SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing (2021, background): “, 2021) have drawn much attention in the NLP and speech communities, due to its strong capability of generalization and efficient usage of largescale data (Devlin et al., 2019; Liu et al., 2019; Yang et al., 2019; Lewis et al., 2020; Chen et al., 2021c; Baevski et al., 2020; Lakhotia et al., 2021; K”
Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentation (2022, background): “The improved quality of speech units discretized based on self-supervised speech representations also allows researchers to apply natural language processing (NLP) techniques on speech, such as spoken generative language modeling [9, 26], and emotion conversion cast as a unit-to-unit translation tas”
Generative Spoken Dialogue Language Modeling (2022, methodology): “An interesting aspect of this procedure is that it can capture aspects of speech that are typically not available in written transcriptions and can therefore model prosody and intonation (Kharitonov et al., 2021), or non verbal vocalizations typical of emotional speech (Kreuk et al., 2021).”
Textless Speech-to-Speech Translation on Real Data (2021, methodology): “We further reduce the full orig-unit sequence by removing repeating units (Lakhotia et al., 2021; Lee et al., 2021; Kharitonov et al., 2021; Kreuk et al., 2021).”
Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervision (2023, methodology): “These methods represent continuous audio waveforms as sequences of tokens from a ﬁnite vocabulary, casting speech generation as a language modeling task.”
AudioGen: Textually Guided Audio Generation (2022, methodology): “…k-means and later perform language modeling (Lakho-2 tia et al., 2021; Kharitonov et al., 2022a; Borsos et al., 2022), multi-stream processing (Kharitonov et al., 2022b), speech emotion conversion (Kreuk et al., 2022), spoken dialuge (Nguyen et al., 2022), and speech-to-speech translation…”
Simple and Controllable Music Generation (2023, methodology): “Recently, Agostinelli et al. [2023] proposed representing music using multiple streams of “semantic tokens” and “acoustic tokens”. Then, they applied a cascade of transformer decoders conditioned on a textual-music joint representation [Huang et al., 2022]. Donahue et al. [2023] followed a similar m”
GAN You Hear Me? Reclaiming Unconditional Speech Synthesis from Diffusion Models (2022, background): “HuBERT [18] clusters or clustered spectrogram features), similar to how a language model is trained on text [19, 20].”
Self-Supervised Speech Representation Learning: A Review (2022, background): “One workaround is two-stage training [77], [332];”
Textually Pretrained Speech Language Models (2023, methodology): “Such a modeling framework additionally allows for capturing and modeling prosodic features [Kharitonov et al., 2021], as well as speaker identity [Borsos et al., 2022], or even natural dialogues [Nguyen et al., 2022].”
Generative Pre-training for Speech with Flow Matching (2023, methodology): “But the success is mostly on discriminative tasks, applying self-supervised models for generation application tasks is often less intuitive (Polyak et al., 2021) and under-performing (Tsai et al., 2022).”
DUB: Discrete Unit Back-translation for Speech Translation (2023, methodology): “…generative spoken language model (Lakhotia et al., 2021; Kharitonov et al., 2022), enhance speech representation (Chung et al., 2021; Meng et al., 2022; Chen et al., 2022; Wu et al., 2022; Zhang et al., 2022c), and model direct speech-to-speech translation (Lee et al., 2022; In-aguma et al., 2022).”
Improving Speech Prosody of Audiobook Text-To-Speech Synthesis with Acoustic and Textual Contexts (2022, methodology): “Second, we also use speaker-dependent pitch normalization [19] to disentangle speaker information from pitch values.”
An Overview of Affective Speech Synthesis and Conversion in the Deep Learning Era (2022, background): “We further note that several works are investigating explicit prosodic or rhythmic control [24], [25], [26], [27], [28].”
textless-lib: a Library for Textless Spoken Language Processing (2022, methodology): “…across different devices; (ii) in cases where on-the-fly preprocessing is not required (e.g., there is no randomized data augmentation (Kharitonov et al., 2021b)), an alternative is to preprocess the entire dataset in advance. textless-lib provides a tool for preprocessing arbitrary…”