Audio Albert: A Lite Bert for Self-Supervised Learning of Audio Representation

By Po-Han Chi, Pei-Hung Chung, Tsung-Han Wu, Chun-Cheng Hsieh, Shang-Wen Li, Hung-yi Lee. 2020

166 citations recorded by the source corpus.

Self-supervised speech models are powerful speech representation extractors for downstream applications. Recently, larger models have been utilized in acoustic model training to achieve better performance. We propose Audio ALBERT, a lite version of the self-supervised speech representation model. We apply the lightweight representation extractor to two downstream tasks, speaker classification and phoneme classification. We show that Audio ALBERT achieves performance comparable with massive pre-trained networks in the downstream tasks while having 91% fewer parameters. Moreover, we design probing models to measure how much the latent representations can encode the speaker’s and phoneme’s information. We find that the representations encoded in internal layers of Audio ALBERT contain more information for both phoneme and speaker than the last layer, which is generally used for downstream tasks. Our findings provide a new avenue for using self-supervised networks to achieve better performance and efficiency.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

On Generative Spoken Language Modeling from Raw Audio (2021, neutral): “Examples of information include spectrograms (Chung et al., 2019; Wang et al., 2020; Chi et al., 2021; Liu et al., 2020; Chung and Glass, 2020; Liu et al., 2020; Ling et al., 2020; Ling and Liu, 2020), cluster indices (Baevski et al., 2019; Hsu et al., 2021), derived signal processing features…”
TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech (2020, methodology): “TERA with all the three objectives consistently outperforms CPC [1], AALBERT [21], and Mockingjay [20] in comparable experiment settings.”
DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantization (2020, background): “In speech, instead of prediction, we randomly mask temporal slices of acoustic features and attempt to reconstruct them [21, 22, 23, 24, 25, 16].”
Wav2KWS: Transfer Learning From Speech Representations for Keyword Spotting (2021, methodology): “In natural language processing, BERT is pretrained using masked language modeling [28], [29] and then ﬁne-tuned for various downstream tasks [30]–[32].”
Textless Speech Emotion Conversion using Decomposed and Discrete Representations (2021, background): “Examples of information include spectrograms (Wang et al., 2020; Chi et al., 2021; Chung and Glass, 2020), cluster indices (Hsu et al.”
Hubert: How Much Can a Bad Teacher Benefit ASR Pre-Training? (2021, background): “Predictionbased self-supervised learning has gathered increasing interests recently, where a model is tasked to predict the content of the unseen regions [29, 30, 31, 32, 33, 34, 35] or to contrast the target unseen frame with randomly sampled ones [5, 36, 6, 11].”
A Further Study of Unsupervised Pretraining for Transformer Based Speech Recognition (2020, methodology): “Other studies [12, 13, 14, 15, 16, 17] were motivated by NLP and applied similar methods on speech tasks.”
Self-Supervised Speech Representation Learning: A Review (2022, unknown): “AALBERT [100] LS 360 hrs PC LS 100 hrs(13) LS 100 hrs(13) 80(12) hrs”
Unleashing the Power of Edge-Cloud Generative AI in Mobile Networks: A Survey of AIGC Services (2023, unknown): “This process efﬁciency leads to a substantial time saving, which not only results in diminished energy consumption, but also contributes to a reduced carbon footprint associated with maintaining physical storage infrastructures [141].”
A Brief Overview of Unsupervised Neural Speech Representation Learning (2022, background): “For zero-masking (Jiang et al. 2019; Liu et al. 2020; Wang, Tang, and Livescu 2020; Chi et al. 2021; Ling and Liu 2020), we have c1:T = f(x1:T ◦m1:T ) (9) x̂t = g(ct) (10) Lt = ‖x̂t − xt‖1 , (11) where the ◦ operator denotes the Hadamard product, f(·) is typically a transformer encoder or a…”