Bytes Are All You Need: End-to-end Multilingual Speech Recognition and Synthesis with Bytes

By Bo Li, Yu Zhang, Tara N. Sainath, Yonghui Wu, William Chan. 2018

140 citations recorded by the source corpus.

We present two end-to-end models: Audio-to-Byte (A2B) and Byte-to-Audio (B2A), for multilingual speech recognition and synthesis. Prior work has predominantly used characters, sub-words or words as the unit of choice to model text. These units are difficult to scale to languages with large vocabularies, particularly in the case of multilingual processing. In this work, we model text via a sequence of Unicode bytes, specifically, the UTF-8 variable length byte sequence for each character. Bytes allow us to avoid large softmaxes in languages with large vocabularies, and share representations in multilingual models. We show that bytes are superior to grapheme characters over a wide variety of languages in monolingual end-to-end speech recognition. Additionally, our multilingual byte model outperform each respective single language baseline on average by 4.4% relatively. In Japanese-English code-switching speech, our multilingual byte model outperform our monolingual baseline by 38.6% relatively. Finally, we present an end-to-end multilingual speech synthesis model using byte representations which matches the performance of our monolingual baselines.

Subfields: Artificial Intelligence, Natural Language Processing, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Yu Zhang
3. Dong Yu
4. Wei-Ning Hsu
5. Mohit Bansal
6. Tie-Yan Liu
7. Tao Qin
8. Graham Neubig

In Natural Language Processing

1. Michael Auli
2. Graham Neubig
3. Jinyu Li
4. Kyunghyun Cho
5. Yu Zhang
6. Mohit Bansal
7. Luke S. Zettlemoyer
8. Dong Yu

In Speech & Audio

1. Jinyu Li
2. Yu Zhang
3. Wei-Ning Hsu
4. Gabriel Synnaeve
5. Alexei Baevski
6. Lei He
7. Tara N. Sainath
8. Yonghui Wu

In Signal Processing

1. Jinyu Li
2. Yu Zhang
3. Dong Yu
4. Wei-Ning Hsu
5. Tara N. Sainath
6. Shinji Watanabe
7. Gabriel Synnaeve
8. Yonghui Wu

Citations in the corpus

• ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models (2021, background): “…on short-to-medium length text 2Previous work has also developed token-free approaches for specific tasks: Gillick et al. (2016) for span labeling, Li et al. (2019) for speech recognition and synthesis, and many authors for machine translation (Lee et al., 2017; Costa-jussà et al., 2017; Cherry…”
• SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network (2021, neutral): “For example, in multi-lingual training, the knowledge from a high resource language may transfer to a low resource language [20].”
• Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameters (2020, background): “Introduction The use of multilingual ASR systems [1, 2, 3, 4, 5, 6, 7, 8, 9] that simultaneously transcribe multiple languages has recently become popular to increase language coverage, but covering all of the world’s ∼7000 languages is still far ahead.”
• Deep Learning for Audio Signal Processing (2019, methodology): “Transfer learning has been used to boost the performance of ASR systems on low resource languages with data from rich resource languages [75].”
• Learning to Speak Fluently in a Foreign Language: Multilingual Speech Synthesis and Cross-Language Voice Cloning (2019, methodology): “Our work is most similar to [19], which describes a multilingual TTS model based on Tacotron 2 [20] which uses a Unicode encoding “byte” input representation to train a model on one speaker of each of English, Spanish, and Mandarin.”
• Scaling Speech Technology to 1, 000+ Languages (2023, methodology): “…prior work explored various approaches including byte encoding to unify text representations which was evaluated on English, Spanish, and Chinese [Li et al., 2019] Further studies explored other input representations, including phonemes [Zhang et al., 2019] and phonological features [Staib et…”
• Towards Universal Text-to-Speech (2020, methodology): “In [14], based on Tacotron 2 [10], UTF-8 Byte sequence is used as the input of the model, and a fixed speaker embedding for each speaker is used.”
• Code-Switched Speech Synthesis Using Bilingual Phonetic Posteriorgram with Only Monolingual Corpora (2020, methodology): “[24] uses Unicode byte representation for all languages.”
• Multilingual Byte2Speech Text-To-Speech Models Are Few-shot Spoken Language Learners (2021, methodology): “Adversarial training can be also introduced (Zhang et al., 2019).”
• Recent Advances in End-to-End Automatic Speech Recognition (2021, methodology): “set and then training an E2E model with all the data [38, 131, 149, 276, 298].”
• Cross-Lingual Text-to-Speech Using Multi-Task Learning and Speaker Classifier Joint Training (2022, methodology): “Based on the encoder-decoder E2E framework, various multilingual text-to-speech (TTS) approaches have been proposed [7, 8, 9, 10, 11, 12, 13, 14].”
• Neural Machine Translation with Byte-Level Subwords (2019, methodology): “…bytes has previously been used in several natural language processing tasks: part-of-speech tagging and named entity recognition (Gillick et al. 2016), translation(Costa-Juss, Escolano, and Fonollosa 2017), machine reading (Kenter, Jones, and Hewlett 2018) and speech recognition (Li et al. 2019).”
• Phonological Features for 0-shot Multilingual Speech Synthesis (2020, methodology): “An alternative approach is presented in [10], which explores Unicode bytes as a possible input to a multi-speaker, multilingual Tacotron 2.”
• Training Multilingual Pre-trained Language Model with Byte-level Subwords (2021, methodology): “urely of 256 bytes has also been used in many language and speech tasks such as part-of-speech tagging and named entity recognition [24], translation [25], machine reading [26] and speech recognition [27] but these work did not adopt BPE in text encoding. 3 Building Vocabulary with Byte-Level Subwor”