Scaling Laws for Acoustic Models

By Jasha Droppo, Oguz H. Elibol. 2021

32 citations recorded by the source corpus.

There is a recent trend in machine learning to increase model quality by growing models to sizes previously thought to be unreasonable. Recent work has shown that autoregressive generative models with cross-entropy objective functions exhibit smooth power-law relationships, or scaling laws, that predict model quality from model size, training set size, and the available compute budget. These scaling laws allow one to choose nearly optimal hyper-parameters given constraints on available training data, model parameter count, or training computation budget. In this paper, we demonstrate that acoustic models trained with an auto-predictive coding loss behave as if they are subject to similar scaling laws. We extend previous work to jointly predict loss due to model size, to training set size, and to the inherent "irreducible loss" of the task. We find that the scaling laws accurately match model performance over two orders of magnitude in both model size and training set size, and make predictions about the limits of model performance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Scaling & Learning Theory, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Luke S. Zettlemoyer
2. Percy Liang
3. Mohit Bansal
4. Yejin Choi
5. Graham Neubig
6. Jianfeng Gao
7. Noah A. Smith
8. Quoc V. Le

In Computer Vision and Pattern Recognition

1. Luke S. Zettlemoyer
2. Luc Van Gool
3. Mohit Bansal
4. Trevor Darrell
5. Dacheng Tao
6. Xiaogang Wang
7. Ming-Hsuan Yang
8. Cordelia Schmid

In Large Language Models

1. Luke S. Zettlemoyer
2. Yejin Choi
3. Armen Aghajanyan
4. Omer Levy
5. Michael Lewis
6. Stephen Roller
7. Noah A. Smith
8. Jacob Steinhardt

In Scaling & Learning Theory

1. Luke S. Zettlemoyer
2. Jared Kaplan
3. Susan Zhang
4. Armen Aghajanyan
5. Stanislav Fort
6. Jacob Steinhardt

In Signal Processing

1. Alexis Conneau
2. Luke S. Zettlemoyer
3. Catherine Olsson
4. Dario Amodei
5. Wei-Ning Hsu
6. Stanislav Fort
7. Tom B. Brown
8. Omer Levy

Citations in the corpus

• The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models (2022, background): “Regarding safe ML, several recent papers propose extrapolating empirical trends to forecast future ML capabilities (Kaplan et al., 2020; Hernandez et al., 2021; Droppo & Elibol, 2021), partly to avoid unforeseen consequences from ML.”
• Predictability and Surprise in Large Generative Models (2022, background): “) [35], audition [21], transfer from text to programming [36], few-shot adaptation of vision models [54], and more.”
• Scaling Laws for Generative Mixed-Modal Language Models (2023, background): “Additionally, Henighan et al. (2020) and Droppo & Elibol (2021) examined the application of neural scaling laws to generative language models in different modalities, including image generation and acoustic models.”