Generalization in multitask deep neural classifiers: a statistical physics approach
By Tyler Lee, Anthony Ndirango. 2019
21 citations recorded by the source corpus.
A proper understanding of the striking generalization abilities of deep neural networks presents an enduring puzzle. Recently, there has been a growing body of numerically-grounded theoretical work that has contributed important insights to the theory of learning in deep neural nets. There has also been a recent interest in extending these analyses to understanding how multitask learning can further improve the generalization capacity of deep neural nets. These studies deal almost exclusively with regression tasks which are amenable to existing analytical techniques. We develop an analytic theory of the nonlinear dynamics of generalization of deep neural networks trained to solve classification tasks using softmax outputs and cross-entropy loss, addressing both single task and multitask settings. We do so by adapting techniques from the statistical physics of disordered systems, accounting for both finite size datasets and correlated outputs induced by the training dynamics. We discuss the validity of our theoretical results in comparison to a comprehensive suite of numerical experiments. Our analysis provides theoretical support for the intuition that the performance of multitask learning is determined by the noisiness of the tasks and how well their input features align with each other. Highly related, clean tasks benefit each other, whereas unrelated, clean tasks can be detrimental to individual task performance.
Subfields: Artificial Intelligence, Statistical and Nonlinear Physics
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Statistical and Nonlinear Physics
Citations in the corpus
- Continual Learning in the Teacher-Student Setup: Impact of Task Similarity (2021, neutral): “…et al., 2017; Tian, 2017; Du et al., 2018; Soltanolkotabi et al., 2018; Aubin et al., 2018; Saxe et al., 2018; Baity-Jesi et al., 2018; Goldt et al., 2019; Ghorbani et al., 2019; Yoshida & Okada, 2019; Ndirango & Lee, 2019; Gabrié, 2020; Bahri et al., 2020; Zdeborová, 2020; Advani et al., 2020).”
- Multi-Task Learning with Deep Neural Networks: A Survey (2020, background): “…(Baxter, 2000; Ben-David and Borbely, 2008; Zhang, 2015; Lounici et al., 2009), but aside from a small pool of recent work (Shui et al., 2019; Ndirango and Lee, 2019; D’Eramo et al., 2020; Wu et al., 2020; Zhang et al., 2020; Bettgenhäuser et al., 2020), there is a lack of theoretical…”