Token-Level Ensemble Distillation for Grapheme-to-Phoneme Conversion

By Xu Tan, Jun-Wei Gan, Hongzhi Liu, Sheng Zhao, Tao Qin, Tie-Yan Liu. 2019

68 citations recorded by the source corpus.

Grapheme-to-phoneme (G2P) conversion is an important task in automatic speech recognition and text-to-speech systems. Recently, G2P conversion is viewed as a sequence to sequence task and modeled by RNN or CNN based encoder-decoder framework. However, previous works do not consider the practical issues when deploying G2P model in the production system, such as how to leverage additional unlabeled data to boost the accuracy, as well as reduce model size for online deployment. In this work, we propose token-level ensemble distillation for G2P conversion, which can (1) boost the accuracy by distilling the knowledge from additional unlabeled data, and (2) reduce the model size but maintain the high accuracy, both of which are very practical and helpful in the online production system. We use token-level knowledge distillation, which results in better accuracy than the sequence-level counterpart. What is more, we adopt the Transformer instead of RNN or CNN based models to further boost the accuracy of G2P conversion. Experiments on the publicly available CMUDict dataset and an internal English dataset demonstrate the effectiveness of our proposed method. Particularly, our method achieves 19.88% WER on CMUDict dataset, outperforming the previous works by more than 4.22% WER, and setting the new state-of-the-art results.

Subfields: Artificial Intelligence, Efficiency & Compression, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Zhou Zhao
5. Sheng Zhao
6. Yi Ren
7. Jinyu Li
8. Dong Yu

In Efficiency & Compression

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Sheng Zhao
5. Yi Ren
6. Zhou Zhao
7. Dong Yu
8. Yoshua Bengio

In Speech & Audio

1. Xu Tan
2. Tao Qin
3. Sheng Zhao
4. Tie-Yan Liu
5. Yi Ren
6. Zhou Zhao
7. Lei He
8. Jinyu Li

Citations in the corpus

• NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (2023, methodology): “The input text sequence is first converted into a phoneme sequence using grapheme-to-phoneme conversion [36] and then aligned with speech using our internal alignment tool to obtain the phoneme-level duration.”
• A Survey on Neural Speech Synthesis (2021, methodology): “Thus, G2P conversion in this kind of languages is mainly responsible for polyphone disambiguation, which decides the appropriate pronunciation based on the current word context [441, 392, 224, 295, 321, 29, 257].”
• AdaSpeech: Adaptive Text to Speech for Custom Voice (2021, methodology): “…to 16kHz; 2) extract the mel-spectrogram with 12.5ms hop size and 50ms window size following the common practice in Shen et al. (2018); Ren et al. (2019); 3) convert text sequence into phoneme sequence with grapheme-to-phoneme conversion (Sun et al., 2019) and take phoneme as the encoder input.”
• Make-A-Voice: Revisiting Voice Large Language Models as Scalable Multilingual and Multitask Learners (2024, positive): “For text sequence, we tokenize it into the phoneme sequence with an open-source grapheme-to-phoneme conversion tool (Sun et al., 2019).”
• FastSpeech 2: Fast and High-Quality End-to-End Text to Speech (2020, methodology): “To alleviate the mispronunciation problem, we convert the text sequence into the phoneme sequence (Arik et al., 2017; Wang et al., 2017; Shen et al., 2018; Sun et al., 2019) with an open-source grapheme-to-phoneme tool5.”
• ProDiff: Progressive Fast Diffusion Model for High-Quality Text-to-Speech (2022, methodology): “We convert the text sequence into the phoneme sequence with an open-source graphemeto-phoneme conversion tool [47] 3.”
• Adaspeech 2: Adaptive Text to Speech with Untranscribed Data (2021, methodology): “5 ms hop size and 50ms window size following the common practice in [8]; 3) Convert text sequence into phoneme sequence with grapheme-to-phoneme conversion [19].”
• Make-A-Voice: Unified Voice Synthesis With Discrete Representation (2023, methodology): “S 1 model is trained on text-semantic pairs, where we convert the text sequence into the phoneme sequence with an open-source grapheme-to-phoneme conversion tool [37].”
• MultiSpeech: Multi-Speaker Text to Speech with Transformer (2020, methodology): “We convert text into phoneme using grapheme-to-phoneme conversion [20] and take phoneme as the encoder input.”
• Prosospeech: Enhancing Prosody with Quantized Vector Pre-Training in Text-To-Speech (2022, methodology): “We convert the text sequence to the phoneme sequence [20, 1, 2, 21, 4] with an open-source Chinese grapheme-to-phoneme tool 3 .”
• LRSpeech: Extremely Low-Resource Speech Synthesis and Recognition (2020, background): “, “speech" is converted into “s p iy ch"), which is called as grapheme-to-phoneme conversion [36].”
• AdaSpeech 3: Adaptive Text to Speech for Spontaneous Style (2021, methodology): “We then convert each text sequence into phoneme sequence with grapheme-to-phoneme conversion [18].”
• Regeneration Learning: A Learning Paradigm for Data Generation (2023, methodology): “For example, we can convert a text/character/grapheme sequence into a phoneme sequence using the grapheme-to-phoneme conversion model (Sun et al. 2019).”
• Prompttts: Controllable Text-To-Speech With Text Descriptions (2022, methodology): “We utilize a grapheme-to-phoneme conversion tool [15] to convert the input (content prompt) sequence to a phoneme sequence P = [P1, P2, · · · , PN ], where N refers to the length of phoneme sequence, and we map it to a phoneme embedding to feed into Transformer blocks.”
• RALL-E: Robust Codec Language Modeling with Chain-of-Thought Prompting for Text-to-Speech Synthesis (2024, methodology): “The transcriptions are converted into phonemes by a grapheme-to-phoneme tool [27].”
• HiFiSinger: Towards High-Fidelity Neural Singing Voice Synthesis (2020, methodology): “Specifically, we process the music score as follows: 1) We convert the character (e.g., Chinese) or syllable (e.g., English) in lyrics into phoneme using grapheme-to-phoneme conversion (Taylor, 2005; Sun et al., 2019).”
• PortaSpeech: Portable and High-Quality Generative Text-to-Speech (2021, methodology): “We convert the text sequence to the phoneme sequence [2, 25, 29, 30, 35] with an open-source grapheme-to-”
• Knowledge Distillation from Bert in Pre-Training and Fine-Tuning for Polyphone Disambiguation (2019, background): “For alphabetic languages such as English and French, the main challenge of G2P conversion is how to predict the pronunciations of out-of-vocabulary (OOV) words [3, 4, 5].”