Fastpitch: Parallel Text-to-Speech with Pitch Prediction

By Adrian La'ncucki. 2020

415 citations recorded by the source corpus.

We present FastPitch, a fully-parallel text-to-speech model based on FastSpeech, conditioned on fundamental frequency contours. The model predicts pitch contours during inference. By altering these predictions, the generated speech can be more expressive, better match the semantic of the utterance, and in the end more engaging to the listener. Uniformly increasing or decreasing pitch with FastPitch generates speech that resembles the voluntary modulation of voice. Conditioning on frequency contours improves the overall quality of synthesized speech, making it comparable to state-of-the-art. It does not introduce an overhead, and FastPitch retains the favorable, fully-parallel Transformer architecture, with over 900× real-time factor for mel-spectrogram synthesis of a typical utterance.

Subfields: Artificial Intelligence, Model Architectures, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Speech & Audio

In Signal Processing

Citations in the corpus

A Survey on Neural Speech Synthesis (2021, methodology): “FastPitch [177] improves FastSpeech by using pitch information as decoder input, which shares similar idea of variance predictor in FastSpeech 2.”
Parallel Tacotron: Non-Autoregressive and Controllable TTS (2020, background): “Some of these models train their explicit duration model to predict groundtruth phoneme duration extracted from autoregressive models [15, 18, 22], internal aligner models [17, 19], or external aligner models [16, 23, 24].”
UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generation (2021, methodology): “Vocoders have been employed in various fields such as text-tospeech[1, 2, 3, 4], voice conversion[5], and speech-to-speech translation[6].”
The Multi-Speaker Multi-Style Voice Cloning Challenge 2021 (2021, neutral): “com/M2VoC of speech is beneficial for stylization [4, 5].”
TalkNet 2: Non-Autoregressive Depth-Wise Separable Convolutional Model for Speech Synthesis with Explicit Pitch and Duration Prediction (2021, methodology): “As a result, TalkNet has only 13.2M parameters - almost 2X less than models with similar quality, e.g. the Transformer-based FastSpeech [2, 9], or FastPitch [10].”
FoundationTTS: Text-to-Speech for ASR Customization with Generative Language Model (2023, background): “The first class is to provide additional acoustic or linguistic conditions for the acoustic model except the phoneme sequence input, such as pitch, GST embedding or phoneme embeddings [60, 61, 62], prosodies in different granularities have also been added to TTS in [63, 64].”
StyleTTS: A Style-Based Generative Model for Natural and Diverse Text-to-Speech Synthesis (2022, neutral): “Moreover, because phoneme duration, pitch, and energy are predicted independently from speech, models such as Fast-Speech2 [11] and FastPitch [22] allow fully controllable speech synthesis.”
Recent Developments on Espnet Toolkit Boosted By Conformer (2020, methodology): “For FS2, instead of the quantized pitch and energy prediction, we use the token-average pitch and energy prediction introduced in [23] to avoid overﬁtting.”
RetrieverTTS: Modeling Decomposed Factors for Text-Based Speech Insertion (2022, positive): “For local factors, the non-AR TTS models like FastPitch [18] satisﬁes our requirement, since it learns both text and prosody embeddings.”