textless-lib: a Library for Textless Spoken Language Processing

By Evgeny Kharitonov, Jade Copet, Kushal Lakhotia, Tu Nguyen, Paden Tomasello, Ann Lee, Ali Mamdouh Elkahky, Wei-Ning Hsu, Abdel-rahman Mohamed, Emmanuel Dupoux and 1 more. 2022

41 citations recorded by the source corpus.

Textless spoken language processing is an exciting area of research that promises to extend applicability of the standard NLP toolset onto spoken language and languages with few or no textual resources.Here, we introduce textless-lib, a PyTorch-based library aimed to facilitate research in the area. We describe the building blocks that the library provides and demonstrate its usability by discuss three different use-case examples: (i) speaker probing, (ii) speech resynthesis and compression, and (iii) speech continuation. We believe that textless-lib substantially simplifies research the textless setting and will be handful not only for speech researchers but also for the NLP community at large.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

AudioLM: A Language Modeling Approach to Audio Generation (2022, methodology): “same set of samples using the unit-to-speech synthesis module of GSLM [14] as provided by textless-lib [46].”
AudioGen: Textually Guided Audio Generation (2022, background): “Recent studies suggest quantizing SSL representations using k-means and later perform language modeling (Lakho-2 tia et al., 2021; Kharitonov et al., 2022a; Borsos et al., 2022), multi-stream processing (Kharitonov et al., 2022b), speech emotion conversion (Kreuk et al., 2022), spoken dialuge…”
Make-A-Voice: Unified Voice Synthesis With Discrete Representation (2023, positive): “Generative spoken language modeling (GSLM) [17] with “textless NLP” pioneers the use of language models on discrete speech representations.”
Are Discrete Units Necessary for Spoken Language Modeling? (2022, methodology): “Following [23], we train a speaker classifier in the following way: We randomly split LibriSpeech dev-clean utterances into train/valid/test (80%/10%/10%) sets and train a two-layer Transformer classifier on the sequences of discrete units or continuous features of the utterances.”
EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesis (2023, methodology): “We used the available model in textless-lib [21].”
AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head (2023, background): ““textless NLP” (Kharitonov et al., 2022; Huang et al., 2022c) is proposed to model language directly without any transcription by training autoregressive generative models of low-bitrate audio tokens.”
Textually Pretrained Speech Language Models (2023, background): “Models and quantizers are obtained from the textless-lib [Kharitonov et al., 2022].”