On Using Backpropagation for Speech Texture Generation and Voice Conversion

By Jan Chorowski, Ron J. Weiss, Rif A. Saurous, Samy Bengio. 2017

19 citations recorded by the source corpus.

Inspired by recent work on neural network image generation which rely on backpropagation towards the network inputs, we present a proof-of-concept system for speech texture synthesis and voice conversion based on two mechanisms: approximate inversion of the representation learned by a speech recognition neural network, and on matching statistics of neuron activations between different source and target utterances. Similar to image texture synthesis and neural style transfer, the system works by optimizing a cost function with respect to the input waveform samples. To this end we use a differentiable mel-filterbank feature extraction pipeline and train a convolutional cTC speech recognition network. Our system is able to extract speaker characteristics from very limited amounts of target speaker data, as little as a few seconds, and can be used to generate realistic speech babble or reconstruct an utterance in a different voice.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Speech & Audio

In Signal Processing

Citations in the corpus

Unsupervised Speech Representation Learning Using WaveNet Autoencoders (2019, background): “speaker gender and identity, from phonetic content, properties which are consistent with internal representations learned by speech recognizers [13], [14].”
Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron (2018, methodology): “To perform voice conversion, a model must synthesize an utterance, given only the acoustic signal of that utterance in a different speaker’s voice (Wu et al., 2013; Nakashika et al., 2016; Kinnunen et al., 2017; van den Oord et al., 2017; Chorowski et al., 2017).”
Text-Free Image-to-Speech Synthesis Using Learned Segmental Units (2020, methodology): “Voice conversion has recently seen progress using neural approaches (Hsu et al., 2016, 2017b,c,a; Fang et al., 2018; Chorowski et al., 2018; Chou et al., 2018; Lorenzo-Trueba et al., 2018; Kameoka et al., 2018; Serrà et al., 2019), but the most relevant work to our own is the ZeroSpeech 2019…”