Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representations

By Hyeong-Seok Choi, Juheon Lee, Wan Soo Kim, Jie Hwan Lee, Hoon Heo, Kyogu Lee. 2021

198 citations recorded by the source corpus.

We present a neural analysis and synthesis (NANSY) framework that can manipulate voice, pitch, and speed of an arbitrary speech signal. Most of the previous works have focused on using information bottleneck to disentangle analysis features for controllable synthesis, which usually results in poor reconstruction quality. We address this issue by proposing a novel training strategy based on information perturbation. The idea is to perturb information in the original input signal (e.g., formant, pitch, and frequency response), thereby letting synthesis networks selectively take essential attributes to reconstruct the input signal. Because NANSY does not need any bottleneck structures, it enjoys both high reconstruction quality and controllability. Furthermore, NANSY does not require any labels associated with speech data such as text and speaker information, but rather uses a new set of analysis features, i.e., wav2vec feature and newly proposed pitch feature, Yingram, which allows for fully self-supervised training. Taking advantage of fully self-supervised training, NANSY can be easily extended to a multilingual setting by simply training it with a multilingual dataset. The experiments show that NANSY can achieve significant improvement in performance in several applications such as zero-shot voice conversion, pitch shift, and time-scale modification.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhou Zhao
2. Rongjie Huang
3. Yi Ren
4. Dong Yu
5. Jinglin Liu
6. Chao Weng
7. Dongchao Yang
8. Huadai Liu

In Representation & Self-Supervised Learning

1. Dong Yu
2. Moacir Antonelli Ponti
3. Chao Weng
4. Yi Ren
5. Dongchao Yang
6. Jinzheng He
7. Rongjie Huang
8. Ziyue Jiang

In Speech & Audio

1. Rongjie Huang
2. Jinglin Liu
3. Zhou Zhao
4. Yi Ren
5. Dong Yu
6. Dongchao Yang
7. Chao Weng
8. Huadai Liu

In Signal Processing

1. Rongjie Huang
2. Zhou Zhao
3. Yi Ren
4. Jinglin Liu
5. Dongchao Yang
6. Chao Weng
7. Dong Yu
8. Huadai Liu

Citations in the corpus

• NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (2024, neutral): “Prior works [36, 37, 38] utilize disentangled representation for speech generation, such as speech content from self-supervised pre-trained models [39, 40, 41], fundamental frequency, and timbre, but speech quality is not satisfying.”
• YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone (2021, background): “Voice conversion is yet another task that may benefit from multilingual contexts (Choi et al., 2021).”
• Make-A-Voice: Revisiting Voice Large Language Models as Scalable Multilingual and Multitask Learners (2024, positive): “Building a simple and multitask learning framework has attracted increasing attention in the community: NANSY (Choi et al., 2021a, 2022) is trained in a self-supervised manner that does not require any annotations paired with audio, and efficiently tackles multiple applications after training the…”
• Make-A-Voice: Unified Voice Synthesis With Discrete Representation (2023, background): “NANSY families [6, 7] are trained in a self-supervised manner that does not require any annotations paired with audio.”
• TranSpeech: Speech-to-Speech Translation With Bilateral Perturbation (2022, background): “…models have demonstrated the efficiency in promoting downstream performance: SpeechSplit (Qian et al., 2020), AutoPST (Qian et al., 2021), and NANSY (Choi et al., 2021) perturb the speech variations during the analysis stage to encourage the synthesis stage to use the supplied more stable…”
• NANSY++: Unified Voice Synthesis with Neural Analysis and Synthesis (2022, methodology): “Next, we adopt two self-supervised training methods - information perturbation and bottleneck - to extract a linguistic representation that is disentangled from other representations (Choi et al., 2021a; Qian et al., 2022).”
• A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI (2023, unknown): “The ground truth style features for diffusion model during training are generated by a pretrained style teacher (e.g., GenerSpeech [27] and NANSY [10]) on clean audio.”
• LM-VC: Zero-Shot Voice Conversion via Speech Generation Based on Language Models (2023, methodology): “Some studies [2], [5] also use signal perturbation techniques to alter speech utterances to make it speaker irrelevant before content extraction.”
• HierSpeech++: Bridging the Gap Between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-Shot Speech Synthesis (2023, unknown): “Following Nansy [69], we adopt formant shifting, pitch randomization, and random frequency shaping for speech perturbation.”