Cycle-Consistent Speech Enhancement

By Zhong Meng, Jinyu Li, Yifan Gong, Biing-Hwang Fred Juang. 2018

53 citations recorded by the source corpus.

Feature mapping using deep neural networks is an effective approach for single-channel speech enhancement. Noisy features are transformed to the enhanced ones through a mapping network and the mean square errors between the enhanced and clean features are minimized. In this paper, we propose a cycle-consistent speech enhancement (CSE) in which an additional inverse mapping network is introduced to reconstruct the noisy features from the enhanced ones. A cycle-consistent constraint is enforced to minimize the reconstruction loss. Similarly, a backward cycle of mappings is performed in the opposite direction with the same networks and losses. With cycle-consistency, the speech structure is well preserved in the enhanced features while noise is effectively reduced such that the feature-mapping network generalizes better to unseen data. In cases where only unparalleled noisy and clean data is available for training, two discriminator networks are used to distinguish the enhanced and noised features from the clean and noisy ones. The discrimination losses are jointly optimized with reconstruction losses through adversarial multi-task learning. Evaluated on the CHiME-3 dataset, the proposed CSE achieves 19.60% and 6.69% relative word error rate improvements respectively when using or without using parallel clean and noisy speech data.

Subfields: Artificial Intelligence, Computational Mechanics, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Haizhou Li
3. Berrak Sisman
4. Junichi Yamagishi
5. Sakriani Watiasri Sakti
6. Andros Tjandra
7. Satoshi Nakamura
8. Yifan Gong

In Computational Mechanics

1. Jinyu Li
2. Haizhou Li
3. Junichi Yamagishi
4. Yifan Gong
5. Yong Zhao

In Speech & Audio

1. Jinyu Li
2. Sakriani Watiasri Sakti
3. Satoshi Nakamura
4. Andros Tjandra
5. Haizhou Li
6. Berrak Sisman
7. Junichi Yamagishi
8. Mingyang Zhang

In Signal Processing

1. Jinyu Li
2. Haizhou Li
3. Berrak Sisman
4. Sakriani Watiasri Sakti
5. Satoshi Nakamura
6. Andros Tjandra
7. Yifan Gong
8. Zhong Meng

Citations in the corpus

• VQVAE Unsupervised Unit Discovery and Multi-scale Code2Spec Inverter for Zerospeech Challenge 2019 (2019, methodology): “However, our task is more challenging due to the fact that our generated voice will always have As a future work, we will investigate the use of GAN-based speech enhancement [31] approaches to further improve our results.”
• Conditional Teacher-student Learning (2019, background): “T/S learning [3, 8, 9] and adversarial learning [20, 21, 22, 23, 24] are two effective approaches that can suppress this domain mismatch by adapting a source-domain acoustic model to target-domain speech.”
• An Overview of Voice Conversion and Its Challenges: From Statistical Modeling to Deep Learning (2020, background): “data does not exist, such as image manipulation and synthesis [194], [196], [200]–[204], speech enhancement [205], speech recognition [206], speech synthesis [207], [208], and music”