Parrotron: An End-to-End Speech-to-Speech Conversion Model and its Applications to Hearing-Impaired Speech and Speech Separation

By Fadi Biadsy, Ron J. Weiss, Pedro J. Moreno Mengibar, Dimitri Kanvesky, Ye Jia. 2019

125 citations recorded by the source corpus.

We describe Parrotron, an end-to-end-trained speech-to-speech conversion model that maps an input spectrogram directly to another spectrogram, without utilizing any intermediate discrete representation. The network is composed of an encoder, spectrogram and phoneme decoders, followed by a vocoder to synthesize a time-domain waveform. We demonstrate that this model can be trained to normalize speech from any speaker regardless of accent, prosody, and background noise, into the voice of a single canonical target speaker with a fixed accent and consistent articulation and prosody. We further show that this normalization model can be adapted to normalize highly atypical speech from a deaf speaker, resulting in significant improvements in intelligibility and naturalness, measured via a speech recognizer and listening tests. Finally, demonstrating the utility of this model on other speech tasks, we show that the same model architecture can be trained to perform a speech separation task

Subfields: Artificial Intelligence, Natural Language Processing, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yu Zhang
2. Shinji Watanabe
3. Ye Jia
4. Junichi Yamagishi
5. Heiga Zen
6. Xu Tan
7. Yonghui Wu
8. Tao Qin

In Natural Language Processing

1. Yu Zhang
2. Ye Jia
3. Heiga Zen
4. Yonghui Wu
5. Shinji Watanabe
6. Ron J. Weiss
7. Mohammad Norouzi
8. William Chan

In Speech & Audio

1. Yu Zhang
2. Shinji Watanabe
3. Ye Jia
4. Yonghui Wu
5. Tomoki Toda
6. Ron J. Weiss
7. Junichi Yamagishi
8. Heiga Zen

In Signal Processing

1. Yu Zhang
2. Shinji Watanabe
3. Heiga Zen
4. Junichi Yamagishi
5. Ye Jia
6. Yonghui Wu
7. Ron J. Weiss
8. Xu Tan

Citations in the corpus

• WaveGrad: Estimating Gradients for Waveform Generation (2020, background): “Deep generative models have revolutionized speech synthesis (Oord et al., 2016; Sotelo et al., 2017; Wang et al., 2017; Biadsy et al., 2019; Jia et al., 2019).”
• Direct speech-to-speech translation with a sequence-to-sequence model (2019, methodology): “The proposed model resembles recent sequence-to-sequence models for voice conversion, the task of recreating an utterance in another person’s voice [22–24].”
• Semi-Supervised Speaker Adaptation for End-to-End Speech Synthesis with Pretrained Models (2020, methodology): “We use the CER for finding the deletion of speech, repetition of speech and mispronounce as deletion error, insertion error and substitution error, respectively [35].”
• Many-to-Many Voice Transformer Network (2020, methodology): “proposed an end-to-end VC model called “Parrotron,” which uses a multitask learning strategy to train the encoder and decoder along with an ASR model [46].”
• Zero-Shot Voice Style Transfer with Only Autoencoder Loss (2019, background): “Text transcriptions are introduced to assist the learning of the latent code (Xie et al., 2016; Saito et al., 2018; Biadsy et al., 2019), but we will focus on voice conversion without text transcriptions, which is more flexible for low-resourced languages.”
• Unsupervised Speech Decomposition via Triple Information Bottleneck (2020, background): “However, Parrotron still requires text transcriptions to label the phone representation, as well as to generate the synthetic parallel dataset.”
• Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modeling (2020, methodology): “Similar models have been used for more complicated problems, like direct speech-to-speech translation (Jia et al., 2019), speech conversion (Biadsy et al., 2019), and speech enhancement (Ding et al., 2020).”