Hubert: How Much Can a Bad Teacher Benefit ASR Pre-Training?

By Wei-Ning Hsu, Yao-Hung Hubert Tsai, Benjamin Bolte, Ruslan Salakhutdinov, Abdel-rahman Mohamed. 2021

184 citations recorded by the source corpus.

Compared to vision and language applications, self-supervised pre-training approaches for ASR are challenged by three unique problems: (1) There are multiple sound units in each input utterance, (2) With audio-only pre-training, there is no lexicon of sound units, and (3) Sound units have variable lengths with no explicit segmentation. In this paper, we propose the Hidden-Unit BERT (HUBERT) model which utilizes a cheap k-means clustering step to provide aligned target labels for pre-training of a BERT model. A key ingredient of our approach is applying the predictive loss over the masked regions only. This allows the pre-training stage to benefit from the consistency of the unsupervised teacher rather that its intrinsic quality. Starting with a simple k-means teacher of 100 cluster, and using two iterations of clustering, the HUBERT model matches the state-of-the-art wav2vec 2.0 performance on the ultra low-resource Libri-light 10h, 1h, 10min supervised subsets.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Speech & Audio

In Signal Processing

Citations in the corpus

Unsupervised Speech Recognition (2021, methodology): “Hsu et al. (2021b) leverages automatically discovered speech units to learn representations with a BERT-like masked prediction objective and achieves comparable results with models based on contrastive learning.”
AudioLM: A Language Modeling Approach to Audio Generation (2022, methodology): “For GSLM resynthesis, we use a vocabulary of 200 tokens derived from HuBERT representations [57] which was found to provide the lowest resynthesis error [14].”
Speech Resynthesis from Discrete Disentangled Self-Supervised Representations (2021, methodology): “For HuBERT we used a BASE 12 transformerlayer model trained for two iterations [4] on 960 hours of LibriSpeech corpus [46].”
Text-Free Prosody-Aware Generative Spoken Language Modeling (2021, methodology): “As described in Section 3.1, we use HuBERT-based unit representations.”
SUPERB: Speech processing Universal PERformance Benchmark (2021, methodology): “Discriminative modeling for SSL studied here include CPC [11, 34], wav2vec [12], vq-wav2vec [13], wav2vec 2.0 [14] and HuBERT [35].”
WENETSPEECH: A 10000+ Hours Multi-Domain Mandarin Corpus for Speech Recognition (2021, neutral): “0 [18], Hubert [19], and wav2vec-U [20], etc.”
data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language (2022, methodology): “, 2021), or the learning of a vocabulary of discrete speech units (Baevski et al., 2020b; Hsu et al., 2021).”
Text-Free Image-to-Speech Synthesis Using Learned Segmental Units (2020, background): “Prior work addresses inferring linguistic content such as phones from the learned representations (Baevski et al., 2020; Kharitonov et al., 2020; Hsu et al., 2021).”
Automatic speaker verification spoofing and deepfake detection using wav2vec 2.0 and data augmentation (2022, unknown): “Two particularly popular approaches, HuBERT and wav2vec 2.0, have been applied to automatic speech recognition [12, 13], mispronunciation detection [36, 37], speaker recognition [38, 39] and emotion recognition [40].”
SSAST: Self-Supervised Audio Spectrogram Transformer (2021, methodology): “Finally, we pre-train the model with joint discriminative and generative objectives, which is also novel in the audio and speech domain and only has been explored in (Pascual et al. 2019; Jiang et al. 2020; Ravanelli et al. 2020).”
Layer-Wise Analysis of a Self-Supervised Speech Representation Model (2021, background): “Some very recent work has explored the phonetic, paralinguistic, and semantic content in SSL models using classiﬁer probes [10, 16, 21, 22] and relationships be-tween models with different training objectives and architectures [23].”
Performance-Efficiency Trade-Offs in Unsupervised Pre-Training for Speech Recognition (2021, methodology): “SEW models can function as direct replacement of W2V2 models, including in recent work (Hsu et al., 2020; 2021; Xu et al., 2020a).”
Textless Speech Emotion Conversion using Decomposed and Discrete Representations (2021, methodology): “The content encoder Ec is a HuBERT model (Hsu et al., 2021) pre-trained on the LibriSpeech corpus (Panayotov et al.”
Recent Advances in End-to-End Automatic Speech Recognition (2021, background): “[31] W.-N. Hsu, Y.-H. H. Tsai, B. Bolte, R. Salakhutdinov, and A. Mohamed, “HuBERT: How much can a bad teacher benefit ASR pre-training?” in Proc.”
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction (2022, unknown): “0 (Ling & Liu, 2020), HuBERT (Hsu et al., 2021c;a)) are mostly built entirely on audio.”
Open-Source Embedding Models: A Comprehensive Survey of Techniques, Benchmarks, and Applications (2026, unknown): “In audio, models such as Wav2Vec 2.0 [22] and HuBERT [23] generate robust speech embeddings, enabling tasks like classification, speaker identification, and audio-text retrieval (e.g., CLAP and AudioCLIP).”