TTS Skins: Speaker Conversion via ASR

By Adam Polyak, Lior Wolf, Yaniv Taigman. 2019

30 citations recorded by the source corpus.

We present a fully convolutional wav-to-wav network for converting between speakers' voices, without relying on text. Our network is based on an encoder-decoder architecture, where the encoder is pre-trained for the task of Automatic Speech Recognition, and a multi-speaker waveform decoder is trained to reconstruct the original signal in an autoregressive manner. We train the network on narrated audiobooks, and demonstrate multi-voice TTS in those voices, by converting the voice of a TTS robot.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

Speech Resynthesis from Discrete Disentangled Self-Supervised Representations (2021, methodology): “These representations are being used together with the identity and prosodic information for voice conversion [24, 25, 26].”
On Generative Spoken Language Modeling from Raw Audio (2021, neutral): “Another line of work use representations from an ASR acoustic model that are combined with identity and prosodic information for voice conversion (Polyak et al., 2020b,a, 2021b).”
High Fidelity Speech Regeneration with Application to Speech Enhancement (2021, methodology): “In addition to differences that arise from the different nature of the task, there are multiple technical differences between the approaches: the generator of [19] is autoregressive, the previous method did not include an identity network and modelled new speakers inaccurately, and lastly the loss t”
Preserving integrity in online social networks (2020, background): “These advances have unleashed potential for harmful applications like face puppeteering [83, 136], speech manipulation [97], face transfer [115], full body manipulation [23], and other more general media manipulations.”
Textless Speech Emotion Conversion using Decomposed and Discrete Representations (2021, background): “Another line of works consists of modifying the parameters of the speech signal (Polyak et al., 2020b,a; Sisman et al., 2020; Robinson et al., 2019; Zhou et al., 2021b), i.e, Speech-to-Speech.”
The DeepFake Detection Challenge Dataset (2020, methodology): “In addition to these face-swapping methods, we also performed audio swapping on some video clips using the TTS Skins voice conversion method detailed in [22].”
Unsupervised Cross-Domain Singing Voice Conversion (2020, methodology): “Our method is inspired by recent work on speech voice conversion [1] and music synthesis [2].”