HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units
By Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, Abdel-rahman Mohamed. 2021
4,844 citations recorded by the source corpus.
Self-supervised approaches for speech representation learning are challenged by three unique problems: (1) there are multiple sound units in each input utterance, (2) there is no lexicon of input sound units during the pre-training phase, and (3) sound units have variable lengths with no explicit segmentation. To deal with these three problems, we propose the Hidden-Unit BERT (HuBERT) approach for self-supervised speech representation learning, which utilizes an offline clustering step to provide aligned target labels for a BERT-like prediction loss. A key ingredient of our approach is applying the prediction loss over the masked regions only, which forces the model to learn a combined acoustic and language model over the continuous inputs. HuBERT relies primarily on the consistency of the unsupervised clustering step rather than the intrinsic quality of the assigned cluster labels. Starting with a simple k-means teacher of 100 clusters, and using two iterations of clustering, the HuBERT model either matches or improves upon the state-of-the-art wav2vec 2.0 performance on the Librispeech (960 h) and Libri-light (60,000 h) benchmarks with 10 min, 1 h, 10 h, 100 h, and 960 h fine-tuning subsets. Using a 1B parameter model, HuBERT shows up to 19% and 13% relative WER reduction on the more challenging dev-other and test-other evaluation subsets.12
Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Speech & Audio, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Representation & Self-Supervised Learning
- Wei-Ning Hsu
- James R. Glass
- Abdel-rahman Mohamed
- Yu-An Chung
- Jinyu Li
- David F. Harwath
- Emmanuel Dupoux
- Alexis Conneau
In Speech & Audio
In Signal Processing
Citations in the corpus
- Robust Speech Recognition via Large-Scale Weak Supervision (2022, neutral): “…2020) • facebook/wav2vec2-large-960h (Baevski et al., 2020) • facebook/hubert-large-ls960-ft (Hsu et al., 2021a) • facebook/hubert-xlarge-ls960-ft (Hsu et al., 2021a) • facebook/s2t-medium-librispeech-asr (Wang et al., 2020a) • facebook/s2t-large-librispeech-asr (Wang et al., 2020a) •…”
- Unispeech-Sat: Universal Speech Representation Learning With Speaker Aware Pre-Training (2021, positive): “We integrate our proposed training method in the Hu-BERT framework [6], and conduct experiment on Speech processing Universal PERformance Benchmark (SUPERB) [15].”
- Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (2023, positive): “Self-supervised learning has been widely investigated in the field of speech understanding [29], [30], [31] and speech-to-speech generation [32], [33].”
- High Fidelity Neural Audio Compression (2022, background): “Similarly, Lakhotia et al. (2021); Kharitonov et al. (2021) demonstrated that self-supervised learning methods for speech (e.g., HuBERT (Hsu et al., 2021)), can be quantized and used for conditional and unconditional speech generation.”
- NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (2024, methodology): “SpeechTokenizer [42] uses HuBERT [43] for semantic distillation, aiming to render the first-layer RVQ representation as semantic information.”
- Audiobox: Unified Audio Generation with Natural Language Prompts (2023, methodology): “WER with ∗ are computed using HuBERT-L ASR that is not comparable with the other numbers.”
- UniAudio: An Audio Foundation Model Toward Universal Audio Generation (2023, positive): “Following Huang et al. (2023b), these discrete representations are tokenized by performing K-means clustering (Hsu et al., 2021) over these continuous representations.”
- FunCodec: A Fundamental, Reproducible and Integrable Open-Source Toolkit for Neural Speech Codec (2023, methodology): “In addition to the encoder outputs, we explore three methods to incorporate semantic information into the codec models: where V s denotes semantic tokens, such as frame-aligned phoneme labels and Hubert embeddings [26].”
- RepCodec: A Speech Representation Codec for Speech Tokenization (2023, methodology): “While PNMI defined in Hsu et al. (2021) cannot reflect the quality of semantic tokens for downstream tasks, we believe the tokens of RepCodec still correspond to semantic tokens like phonemes.”
- Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale (2023, methodology): “Large scale in-context learning models: With the advancement in neural codec for speech [22, 12, 67], many recent studies explore token-based language modeling for speech generation.”
- AudioPaLM: A Large Language Model That Can Speak and Listen (2023, positive): “Similar to BERT [Devlin et al., 2018], such encoders may be trained with a masked language model objective for both the multimodal setting as in previously mentioned works and for the unimodal setting [Baevski et al., 2020, Hsu et al., 2021, Chiu et al., 2022].”
- UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocoding (2023, methodology): “Then we resynthesize the speech in test set B from HuBERT and vq-wav2vec tokens and calculate their WERs respectively with Whisper (Radford et al. 2023), a well-known automatic speech recognition (ASR) model.”
- StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models (2023, background): “0 [9], HuBERT [10], and WavLM [11] as discriminators, in conjunction with a novel differentiable duration modeling approach.”
- Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias (2023, methodology): “We use the ASR system from the released HuBERT-Large model [19] to transcribe the generated speech into text.”
- PolyVoice: Language Models for Speech to Speech Translation (2023, methodology): “Semantic units are usually derived from representations produced by speech encoder models like HuBERT (Hsu et al., 2021), mHuBERT (Lee et al., 2021) or w2v-BERT (Chung et al., 2021).”
- VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translation (2023, positive): “However, it performs speech discretization with hidden units from HuBERT, which contains less acoustic information than codec codes, and only lists some audio examples without quantitative results for cross-modal instruction following and speech dialogue tasks.”
- SoundStorm: Efficient Parallel Audio Generation (2023, positive): “…et al., 2020; Hsu et al., 2021; Chung et al., 2021) have provided a low-framerate representation of the underlying signal which remains rich enough after discretization for language models to generate intelligible speech from a specific speaker as a sequence of tokens (Lakhotia et al., 2021).”
- NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (2023, methodology): “The ASR model is a CTC-based HuBERT [48] pre-trained on Librilight [49] and fine-tuned on the 960 hours training set of LibriSpeech.”
- Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling (2023, positive): “To verify the quality of generated speech, we first utilize the ASR system from the released HuBERT-Large model [Hsu et al., 2021] to recognize it into text.”