Robust Speech Recognition via Large-Scale Weak Supervision
By Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, Ilya Sutskever. International Conference on Machine Learning, 2022
7,824 citations recorded by the source corpus.
We study the capabilities of speech processing systems trained simply to predict large amounts of transcripts of audio on the internet. When scaled to 680,000 hours of multilingual and multitask supervision, the resulting models generalize well to standard benchmarks and are often competitive with prior fully supervised results but in a zero-shot transfer setting without the need for any fine-tuning. When compared to humans, the models approach their accuracy and robustness. We are releasing models and inference code to serve as a foundation for further work on robust speech processing.
Subfields: Artificial Intelligence, Large Language Models, Speech & Audio, Signal Processing
View Paper: https://arxiv.org/abs/2212.04356
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Large Language Models
- Michael Auli
- Alexei Baevski
- Gabriel Synnaeve
- Alexis Conneau
- Ankur Bapna
- Yossi Adi
- Florian Tramèr
- Nicholas Carlini
In Speech & Audio
- Jinyu Li
- Yu Zhang
- Wei-Ning Hsu
- Tara N. Sainath
- Shinji Watanabe
- Yonghui Wu
- Gabriel Synnaeve
- Yossi Adi
In Signal Processing
Citations in the corpus
- RepCodec: A Speech Representation Codec for Speech Tokenization (2023, methodology): “This metric is evaluated using the Whisper large-v2 ASR model (Radford et al., 2023).”
- AudioPaLM: A Large Language Model That Can Speak and Listen (2023, neutral): “, 2022b] in the vision domain, and Whisper [Radford et al., 2022] in the audio domain.”
- UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocoding (2023, methodology): “Then we resynthesize the speech in test set B from HuBERT and vq-wav2vec tokens and calculate their WERs respectively with Whisper (Radford et al. 2023), a well-known automatic speech recognition (ASR) model.”
- VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translation (2023, neutral): “Whisper (Radford et al., 2022) also employs an encoder-decoder framework trained with large-scale supervised data, but focuses on speech recognition and translation tasks.”
- SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models (2023, methodology): “Content preservation is evaluated by computing the WER through transcribing the resynthesized speech using the Whisper en-medium model (Radford et al., 2023).”
- WhisperX: Time-Accurate Speech Transcription of Long-Form Audio (2023, background): “Whisper [9] utilises this rich source of data to another scale.”
- The future landscape of large language models in medicine (2023, background): “Combining the potential of LLMs in the processing and production of both written and spoken language 43 could result in automated dictation or prompt-triggered chart review.”
- Discrete Audio Representation as an Alternative to Mel-Spectrograms for Speaker and Speech Recognition (2023, background): “Transformer-based models are increasingly gaining prominence[3], driven by the interest for a foundational audio model capable of both speech comprehension and generation.”
- Artificial Intelligence-Enabled Intelligent Assistant for Personalized and Adaptive Learning in Higher Education (2023, methodology): “OpenAI's Generative Pretrained Transformer 3.5 (GPT-3.”
- Qwen2-Audio Technical Report (2024, methodology): “Different from Qwen-Audio, the initialization of the audio encoder of Qwen2-Audio is based on the Whisper-large-v3 model (Radford et al., 2023).”
- Foundation models for generalist medical artificial intelligence (2023, methodology): “A GMAI solution can draw from recent advances in speech-to-text models 28 , specializing techniques for medical applications.”
- Speech Recognition via Large-Scale Weak Supervision (positive): “…y 1 ,..., y m EOS (cid:105) ) , assuming special start of sequence and end of sequence tokens (cid:104) sos (cid:105) and (cid:104) eos (cid:105) and each y i is a character; for English we might choose the set: This architecture is also used in the Whisper model from OpenAI (Radford et al., 2023).”
- Seed-TTS: A Family of High-Quality Versatile Speech Generation Models (2024, methodology): “For WER, we employ Whisper-large-v3 [Radford et al., 2023] and Paraformer-zh [Gao et al., 2023b] as the automatic speech recognition (ASR) engines for English and Mandarin, respectively.”
- F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching (2024, methodology): “For WER, we employ Whisper-large-v3 [66] to transcribe English and Paraformer-zh [67] for Chinese, following [39].”
- MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark (2024, neutral): “…results using two types of captions: weak captions (generated using EnCLAP (Kim et al., 2024) for sounds, MuLLaMa for music, and Whis-per base (Radford et al., 2023) for speech transcripts) and strong, detailed captions (generated using Qwen2-Audio-Instruct with prompts detailed in Appendix…”
- LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPT (2023, background): “The third group, such as Whisper (Radford et al., 2022), USM (Zhang et al., 2023b), and Pengi (Deshmukh et al., 2023), focus on converting continuous speech features into text, but they cannot support audio generation tasks .”
- AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head (2023, background): “Task Input Output Domain Model Speech Recognition Audio Text Speech Whisper (Radford et al., 2022) Speech Translation Audio Text Speech MultiDecoder (Dalmia et al.”
- MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities (2023, neutral): “Our MM-Vet is most related to the concurrent evaluation studies [26, 52, 44, 76] P r o p o r t i o n ( % ) 28.4 17.0 11.9 6.4 5.5 5.5 5.0 4.1 3.7 3.7 3.2 1.8 1.4 0.9 0.9 0.5”
- i-Code V2: An Autoregressive Generation Framework over Vision, Language, and Speech Data (2023, background): “Whisper (Radford et al., 2022) A recently developed speech recognition system that is pretrained on 680K hours labeled speech-text transcript with multitask-supervision.”