PnG BERT: Augmented BERT on Phonemes and Graphemes for Neural TTS

By Ye Jia, Heiga Zen, Jonathan Shen, Yu Zhang, Yonghui Wu. 2021

100 citations recorded by the source corpus.

This paper introduces PnG BERT, a new encoder model for neural TTS. This model is augmented from the original BERT model, by taking both phoneme and grapheme representations of text as input, as well as the word-level alignment between them. It can be pre-trained on a large text corpus in a self-supervised manner, and fine-tuned in a TTS task. Experimental results show that a neural TTS model using a pre-trained PnG BERT as its encoder yields more natural prosody and more accurate pronunciation than a baseline model using only phoneme input with no pre-training. Subjective side-by-side preference evaluations show that raters have no statistically significant preference between the speech synthesized using a PnG BERT and ground truth recordings from professional speakers.

Subfields: Artificial Intelligence, Model Architectures, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Sheng Zhao
5. Lei He
6. Frank K. Soong
7. Ye Jia
8. Cong Han

In Model Architectures

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Chenpeng Du
5. K. Yu
6. Lei He
7. Sheng Zhao
8. Nima Mesgarani

In Speech & Audio

1. Xu Tan
2. Lei He
3. Sheng Zhao
4. Tao Qin
5. Tie-Yan Liu
6. Yanqing Liu
7. Frank K. Soong
8. Chenpeng Du

Citations in the corpus

• AudioPaLM: A Large Language Model That Can Speak and Listen (2023, neutral): “These utterances were obtained by synthesizing the ground-truth translated text with a high-quality TTS system which was modified to enable voice transfer [Jia et al., 2021, 2022c].”
• StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models (2023, methodology): “Several recent works have advanced towards human-level TTS [3, 4, 5] using techniques like BERT pre-training [4, 40, 7] and E2E training [32, 5] with differentiable duration modeling [41, 42].”
• VQTTS: High-Fidelity Text-to-Speech Synthesis with Self-Supervised VQ Acoustic Feature (2022, background): “In addition, [17, 18] introduces syntactic graph and [19, 20] introduces word embeddings to TTS models.”
• A Survey on Neural Speech Synthesis (2021, methodology): “LowResource Self-Supervised Training [51, 368, 433, 78, 140, 346, 197, 352, 71]”
• NaturalSpeech: End-to-End Text-to-Speech Synthesis With Human-Level Quality (2022, background): “Previous works [44] conduct pre-training in character/word level and apply the pre-trained model to phoneme encoder, which will cause inconsistency, and the works [45] directly using phoneme pre-training will suffer from limited capacity due to too small size of phoneme vocabulary.”
• CVSS Corpus and Massively Multilingual Speech-to-Speech Translation (2022, neutral): “It synthesizes speech as natural as professional human speakers (Jia et al., 2021).”
• Mixed-Phoneme BERT: Improving BERT with Mixed Phoneme and Sup-Phoneme Representations for Text to Speech (2022, methodology): “Some works [22, 23] attempt to enhance the TTS phoneme encoder with character information directly rather than introducing an auxiliary BERT model.”
• PHONEME-LEVEL BERT FOR ENHANCED PROSODY OF TEXT-TO-SPEECH WITH GRAPHEME PREDICTIONS (2023, methodology): “We employ the whole word masking and follow previous works [8, 9, 12] where the phoneme tokens of 15% of graphemes in each sequence are selected to be masked at random.”
• Translatotron 2: Robust direct speech-to-speech translation (2021, methodology): “We modified the PnG NAT (Jia et al., 2021) TTS model by incorporating a separately trained speaker encoder (Wan et al., 2018) in the same way as Jia et al. (2018), and trained it on the LibriTTS corpus (Zen et al., 2019).”
• Translatotron 2: High-quality direct speech-to-speech translation with voice preservation (2021, methodology): “We modified the PnG NAT (Jia et al., 2021; Shen et al., 2020) TTS model by incorporating a separately trained speaker encoder (Wan et al., 2018) in the same way as in Jia et al. (2018), and trained it on the LibriTTS corpus (Zen et al., 2019).”
• StyleTTS: A Style-Based Generative Model for Natural and Diverse Text-to-Speech Synthesis (2022, unknown): “T EXT-TO-SPEECH (TTS), also known as speech synthesis, has made significant strides with the advent of deep learning, producing increasingly human-like synthetic speech [1], [2], [3].”
• HierSpeech: Bridging the Gap between Text and Speech by Hierarchical Variational Inference using Self-supervised Representations for Speech Synthesis (2022, unknown): “Although the self-supervised representations of text were previously studied for TTS (Jia et al., 2021), to the best of our knowledge, this is the first study that involves the investigation of self-supervised speech representations for single-stage end-to-end TTS.”
• Leveraging unsupervised and weakly-supervised data to improve direct speech-to-speech translation (2022, methodology): “We used a multilingual multispeaker PnG NAT model [51, 52] to synthesize the source text from the MT dataset into speech, and used the same English TTS model as in [9] to synthesize the targets.”