Towards Universal Speech Discrete Tokens: A Case Study for ASR and TTS

By Yifan Yang, Feiyu Shen, Chenpeng Du, Ziyang Ma, K. Yu, Daniel Povey, Xie Chen. 2023

48 citations recorded by the source corpus.

Self-supervised learning (SSL) proficiency in speech-related tasks has driven research into utilizing discrete tokens for speech tasks like recognition and translation, which offer lower storage requirements and great potential to employ natural language processing techniques. However, these studies, mainly single-task focused, faced challenges like overfitting and performance degradation in speech recognition tasks, often at the cost of sacrificing performance in multi-task scenarios. This study presents a comprehensive comparison and optimization of discrete tokens generated by various leading SSL models in speech recognition and synthesis tasks. We aim to explore the universality of speech discrete tokens across multiple speech tasks. Experimental results demonstrate that discrete tokens achieve comparable results against systems trained on FBank features in speech recognition tasks and outperform mel-spectrogram features in speech synthesis in subjective and objective metrics. These findings suggest that universal discrete tokens have enormous potential in various speech-related tasks. Our work is open-source and publicly available at https://github.com/k2-fsa/icefall.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Shujie Liu
3. Yu Wu
4. Sanyuan Chen
5. Furu Wei
6. Long Zhou
7. Ziyang Ma
8. Zi-Hua Zhang

Citations in the corpus

• BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data (2024, background): “Anyone who feels that if so many more students whom we haven’t actually admitted are sitting in on the course than ones we have that the room had to be changed, then probably auditors will have to be excluded, is likely to agree that the curriculum needs revision.”
• LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPT (2023, methodology): “Our model adopts a decoder-only Transformer framework, which offers enhanced simplicity and generality compared to the Transformer encoder-encoder framework.”
• SpeechLM: Enhanced Speech Pre-Training With Unpaired Textual Data (2022, background): “Recent work by [34] has highlighted the importance of the quality of hidden units for the performance of self-supervised pre-training models.”