SpeechX: Neural Codec Language Model as a Versatile Speech Transformer

By Xiaofei Wang, Manthan Thakker, Zhuo Chen, Naoyuki Kanda, Sefik Emre Eskimez, Sanyuan Chen, Min Tang, Shujie Liu, Jinyu Li, Takuya Yoshioka. 2023

123 citations recorded by the source corpus.

Recent advancements in generative speech models based on audio-text prompts have enabled remarkable innovations like high-quality zero-shot text-to-speech. However, existing models still face limitations in handling diverse audio-text speech generation tasks involving transforming input speech and processing audio captured in adverse acoustic conditions. This paper introduces SpeechX, a versatile speech generation model capable of zero-shot TTS and various speech transformation tasks, dealing with both clean and noisy signals. SpeechX combines neural codec language modeling with multi-task learning using task-dependent prompting, enabling unified and extensible modeling and providing a consistent way for leveraging textual input in speech enhancement and transformation tasks. Experimental results show SpeechX's efficacy in various tasks, including zero-shot TTS, noise suppression, target speaker extraction, speech removal, and speech editing with or without background noise, achieving comparable or superior performance to specialized models across tasks.

Subfields: Artificial Intelligence, Large Language Models, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Sheng Zhao
3. Tao Qin
4. Jinyu Li
5. Shujie Liu
6. Tie-Yan Liu
7. Yanqing Liu
8. Rongjie Huang

In Large Language Models

1. Furu Wei
2. Jinyu Li
3. Jiang Bian
4. Jingren Zhou
5. Dongchao Yang
6. Shiliang Zhang
7. Rongjie Huang
8. Sanyuan Chen

In Speech & Audio

1. Xu Tan
2. Sheng Zhao
3. Rongjie Huang
4. Yanqing Liu
5. Jinyu Li
6. Shujie Liu
7. Dongchao Yang
8. Songxiang Liu

Citations in the corpus

• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (2023, neutral): “SpeechX [71] extended VALL-E with multi-task learning, showcasing its versatility in zero-shot TTS, noise suppression, target speaker extraction, speech removal, and speech editing tasks.”
• VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers (2024, background): “SpeechX [Wang et al., 2023c] extended VALL-E with multi-task learning, demonstrating efficacy in zero-shot TTS, noise suppression, target speaker extraction, speech removal, and speech editing tasks.”
• BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data (2024, methodology): “From BASE TTS’s strong performance on English and Spanish, we caught a first glimpse of a multilingual TTS approach that achieves high expressiveness, adapts to textual clues, is data efficient, uses only public domain data, and works for streaming TTS usecases such as voicing LLM outputs.”
• UniAudio: An Audio Foundation Model Toward Universal Audio Generation (2023, background): “Several recent works are able to handle up to four audio generation tasks, but most of them did not take the universality with respect to tasks as a core design objective Wang et al. (2023c); Maiti et al. (2023).”
• Make-A-Voice: Revisiting Voice Large Language Models as Scalable Multilingual and Multitask Learners (2024, neutral): “Wang et al. (2023b) combine neural codec language modeling with multitask learning using task-dependent prompting, capable of zero-shot TTS and various speech transformation tasks, dealing with clean and noisy signals.”
• Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models (2023, background): “Many works (Wang et al., 2023d; Maiti et al., 2023; Rubenstein et al.; Wang et al., 2023e; Nachmani et al., 2023) unify data format and tasks by directly feeding speech representation (Nachmani et al., 2023) or encoding continuous speech signals as discrete codes (Défossez et al., 2022; Zeghidour…”