LPCNET: Improving Neural Speech Synthesis through Linear Prediction

By Jean-Marc Valin, Jan Skoglund. 2018

500 citations recorded by the source corpus.

Neural speech synthesis models have recently demonstrated the ability to synthesize high quality speech for text-to-speech and compression applications. These new models often require powerful GPUs to achieve real-time operation, so being able to reduce their complexity would open the way for many new applications. We propose LPCNet, a WaveRNN variant that combines linear prediction with recurrent neural networks to significantly improve the efficiency of speech synthesis. We demonstrate that LPCNet can achieve significantly higher quality than WaveRNN for the same network size and that high quality LPCNet speech synthesis is achievable with a complexity under 3 GFLOPS. This makes it easier to deploy neural synthesis applications on lower-power devices, such as embedded systems and mobile phones.

Subfields: Artificial Intelligence, Natural Language Processing, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Wei-Ning Hsu
6. Junichi Yamagishi
7. Jinyu Li
8. Hung-yi Lee

In Natural Language Processing

1. Junichi Yamagishi
2. Tao Qin
3. Tie-Yan Liu
4. Dong Yu
5. Zhenhua Ling
6. Lei He
7. Frank K. Soong
8. Yu Zhang

In Speech & Audio

1. Xu Tan
2. Wei-Ning Hsu
3. Sheng Zhao
4. Lei He
5. Tao Qin
6. Yossi Adi
7. Tie-Yan Liu
8. Jinyu Li

In Signal Processing

1. Xu Tan
2. Tao Qin
3. Frank K. Soong
4. Tie-Yan Liu
5. Junichi Yamagishi
6. Lei He
7. Wei-Ning Hsu
8. Sheng Zhao

Citations in the corpus

• The State of Sparsity in Deep Neural Networks (2019, background): “…and this property has been leveraged to significantly reduce the cost associated with the deployment of deep neural networks, and to enable the deployment of state-of-the-art models in severely resource constrained environments (Theis et al., 2018; Kalchbrenner et al., 2018; Valin & Skoglund, 2018).”
• WaveGrad: Estimating Gradients for Waveform Generation (2020, background): “Autoregressive models, in particular, have been popular for raw audio generation due to their tractable likelihoods, simple inference procedures, and high fidelity samples (Oord et al., 2016; Mehri et al., 2016; Kalchbrenner et al., 2018; Song et al., 2019a; Valin & Skoglund, 2019).”
• You Do Not Need More Data: Improving End-To-End Speech Recognition by Text-To-Speech Data Augmentation (2020, positive): “We also investigate the influence of waveform generation methods in TTS on final ASR performance by using the Griffin-Lim algorithm [15] and our modification of LPCNet [16].”
• SoundStream: An End-to-End Neural Audio Codec (2021, methodology): “Other neural audio codecs adopt different model architectures, e.g., WaveRNN in LPCNet [7] and WaveGRU in Lyra [8], all targeting speech at low bitrates.”
• High Fidelity Neural Audio Compression (2022, methodology): “Neural based audio codecs have been recently proposed and demonstrated promising results (Kleijn et al., 2018; Valin & Skoglund, 2019b; Lim et al., 2020; Kleijn et al., 2021; Zeghidour et al., 2021; Omran et al., 2022; Lin et al., 2022; Jayashankar et al., 2022; Li et al.; Jiang et al., 2022),…”
• Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis (2023, methodology): “Autoregressive models, like WaveNet, generate audio samples sequentially, conditioning each new sample on all previously generated ones (Mehri et al., 2016; Kalchbrenner et al., 2018; Valin and Skoglund, 2019).”
• A Survey on Neural Speech Synthesis (2021, methodology): “As can be seen, TTS models that use RNNbased autoregressive models [376, 297, 227, 357] are slow in both training and inference, with O(N) computation, where N is the sequence length.”
• Speech Resynthesis from Discrete Disentangled Self-Supervised Representations (2021, methodology): “For comparison, we included other speech codecs in our evaluation: Opus [36] wideband at 9 kbps VBR, Codec2 [52] at 2.4 kbps and LPCNet [32] operating at 1.6 kbps.”
• Enhancing into the Codec: Noise Robust Speech Coding with Vector-Quantized Autoencoders (2021, methodology): “As a final point of comparison we also evaluate a pre-trained LPCNet vocoder [6].”
• End-to-End Neural Speech Coding for Real-Time Communications (2022, positive): “Existing neural audio codecs could mainly be grouped into two categories, generative de-coder models [1, 2, 3, 4, 5, 6] and end-to-end neural audio coding [7, 8, 9, 10].”
• Improving Opus Low Bit Rate Quality with Neural Speech Synthesis (2019, methodology): “Even lower complexity and real-time operation was recently reported using LPCNet [15].”
• A Real-Time Wideband Neural Vocoder at 1.6 kb/s Using LPCNet (2019, methodology): “As an improvement over our previous work [12], Laplace-distributed noise is added to the excitation inside the prediction loop, making the signal and excitation noise consistent with each other, as shown in Fig.”
• End-to-End Adversarial Text-to-Speech (2020, positive): “That makes these models impractical for real-time use, although this can be addressed with careful engineering [28, 58].”
• Multi-Band Melgan: Faster Waveform Generation For High-Quality Text-To-Speech (2020, methodology): “AR models have been recently modified to speed up their inference [4, 5, 6].”
• Neural Source-Filter Waveform Models for Statistical Parametric Speech Synthesis (2019, neutral): “such as glottal waveform modeling [22] and linear-prediction coding [23], [24].”
• Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representations (2021, methodology): “By modeling source and filter components with deep networks, it has been used for applications such as vocoder [49, 23, 45] and acoustic feature generation [25].”
• Sequence to Sequence Neural Speech Synthesis with Prosody Modification Capabilities (2019, methodology): “We use a first order pre-emphasis filter as a noise shaping means proposed in [4].”
• Improving Performance of Seen and Unseen Speech Style Transfer in End-to-end Neural TTS (2021, methodology): “1 illustrates our proposed framework to style transfer for both seen and unseen styles, where we adopt Tacotron 2 [3] as the decoder and employ the Mel LPCNet vocoder [25] to reconstruct the waveforms from Mel-spectrogram.”
• HooliGAN: Robust, High Quality Neural Vocoding (2020, background): “Other works address this issue by reducing parameters and complexity (WaveRNN [7], LPCNet [8], FFTNET [9]) and/or replacing the auto-regressive generation with parallel generation (Parallel WaveNet [10], WaveGlow [11], Clarinet [12],MelGAN [4], Parallel WaveGAN [5]).”