Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages

By Yu Zhang, Wei Han, James Qin, Yongqiang Wang, Ankur Bapna, Zhehuai Chen, Nanxin Chen, Bo Li, Vera Axelrod, Gary Wang and 17 more. 2023

396 citations recorded by the source corpus.

We introduce the Universal Speech Model (USM), a single large model that performs automatic speech recognition (ASR) across 100+ languages. This is achieved by pre-training the encoder of the model on a large unlabeled multilingual dataset of 12 million (M) hours spanning over 300 languages, and fine-tuning on a smaller labeled dataset. We use multilingual pre-training with random-projection quantization and speech-text modality matching to achieve state-of-the-art performance on downstream multilingual ASR and speech-to-text translation tasks. We also demonstrate that despite using a labeled training set 1/7-th the size of that used for the Whisper model, our model exhibits comparable or better performance on both in-domain and out-of-domain speech recognition tasks across many languages.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Scaling Speech Technology to 1, 000+ Languages (2023, methodology): “The resulting models were pre-trained on 1,406 languages which is over four times the number of languages of known prior work (Zhang et al. 2023a; §4.2).”
Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models (2023, background): “…unify data format and tasks by directly feeding speech representation (Nachmani et al., 2023) or encoding continuous speech signals as discrete codes (Défossez et al., 2022; Zeghidour et al., 2022; Zhang et al., 2023c), and treating different human speech tasks as conditional generative tasks.”
Discrete Audio Representation as an Alternative to Mel-Spectrograms for Speaker and Speech Recognition (2023, background): “As the name suggests, the former are designed for audio compression where as the later are derived through clustering intermediate layer representations when training for speech representation learning [4, 5, 6, 7].”