Scaling Effect of Self-Supervised Speech Models
By Jie Pu, Yuguang Yang, Yuguang Yang, Ruirui Li, Oguz H. Elibol, Jasha Droppo. 2021
8 citations recorded by the source corpus.
The success of modern deep learning systems is built on two cornerstones, massive amount of annotated training data and ad-vanced computational infrastructure to support large-scale com-putation. In recent years, the model size of state-of-the-art deep learning systems has rapidly increased and sometimes reached to billions of parameters. Herein we take a close look into this phenomenon and present an empirical study on the scaling effect of model size for self-supervised speech models. In particular, we investigate the quantitative relationship between the model size and the loss/accuracy performance on speech tasks. First, the power-law scaling property between the number of parameters and the L 1 self-supervised loss is verified for speech models. Then the advantage of large speech models in learning effective speech representations is demonstrated in two downstream tasks: i) speaker recognition and ii) phoneme classification. Moreover, it has been shown that the model size of self-supervised speech networks is able to compensate the lack of annotation when there is insufficient training data.
Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Speech & Audio
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Hailey Schoelkopf
- Lintang Sutawika
- USVSN Sai Prashanth
- Shivanshu Purohit
- Quentin Anthony
- Stella Biderman
- Oskar van der Wal
- Eric Hallahan
In Representation & Self-Supervised Learning
Citations in the corpus
- Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling (2023, background): “…in the behavior of trained language models as they scale (Kaplan et al., 2020; Henighan et al., 2020; Hernandez et al., 2021; Mikami et al., 2021; Pu et al., 2021; Sharma & Kaplan, 2020; Ghorbani et al., 2021), but prior work connecting these “Scaling Laws” to the learning dynamics of language…”
- Emergent and Predictable Memorization in Large Language Models (2023, background): “In the overwhelming majority of prior work on scaling laws [Brown et al., 2020, Kaplan et al., 2020, Pu et al., 2021, Mikami et al., 2021, Rae et al., 2021, Black et al., 2022, Scao et al., 2022b, Chowdhery et al., 2022], including scaling studies targeting memorization [Carlini et al., 2022,…”