Disentangled Feature Learning for Real-Time Neural Speech Coding

By Xue Jiang, Xiulian Peng, Yuan Zhang, Yan Lu. 2022

22 citations recorded by the source corpus.

Recently end-to-end neural audio/speech coding has shown its great potential to outperform traditional signal analysis based audio codecs. This is mostly achieved by following the VQ-VAE paradigm where blind features are learned, vector-quantized and coded. In this paper, instead of blind end-to-end learning, we propose to learn disentangled features for real-time neural speech coding. Specifically, more global-like speaker identity and local content features are learned with disentanglement to represent speech. Such a compact feature decomposition not only achieves better coding efficiency by exploiting bit allocation among different features but also provides the flexibility to do audio editing in embedding space, such as voice conversion in real-time communications. Both subjective and objective results demonstrate its coding efficiency and we find that the learned disentangled features show comparable performance on any-to-any voice conversion with modern self-supervised speech representation learning models with far less parameters and low latency, showing the potential of our neural coding framework.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Sheng Zhao
3. Yanqing Liu
4. Tao Qin
5. Tie-Yan Liu
6. Jinyu Li
7. Yichong Leng
8. Dongchao Yang

In Signal Processing

1. Xu Tan
2. Sheng Zhao
3. Yanqing Liu
4. Jinyu Li
5. Yichong Leng
6. Jiang Bian
7. Zeqian Ju
8. Kai Shen

Citations in the corpus

• NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (2024, methodology): “Disen-TF-Codec [44] proposes the disentanglement with content and timbre representation, and applies them for zero-shot voice conversion.”