On the Theory of Transfer Learning: The Importance of Task Diversity
By Nilesh Tripuraneni, Michael I. Jordan, Chi Jin. 2020
252 citations recorded by the source corpus.
We provide new statistical guarantees for transfer learning via representation learning--when transfer is achieved by learning a feature representation shared across different tasks. This enables learning on new tasks using far less data than is required to learn them in isolation. Formally, we consider $t+1$ tasks parameterized by functions of the form $f_j \circ h$ in a general function class $\mathcal{F} \circ \mathcal{H}$, where each $f_j$ is a task-specific function in $\mathcal{F}$ and $h$ is the shared representation in $\mathcal{H}$. Letting $C(\cdot)$ denote the complexity measure of the function class, we show that for diverse training tasks (1) the sample complexity needed to learn the shared representation across the first $t$ training tasks scales as $C(\mathcal{H}) + t C(\mathcal{F})$, despite no explicit access to a signal from the feature representation and (2) with an accurate estimate of the representation, the sample complexity needed to learn a new task scales only with $C(\mathcal{F})$. Our results depend upon a new general notion of task diversity--applicable to models with general tasks, features, and losses--as well as a novel chain rule for Gaussian complexities. Finally, we exhibit the utility of our general framework in several models of importance in the literature.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
Citations in the corpus
- Continual learning: a feature extraction formalization, an efficient algorithm, and fundamental obstructions (2022, background): “Some recent theoretical works [18, 19, 26, 36, 20, 35, 9] provide generalization and sample complexity guarantees for certain formalizations of multi-task learning based on the existence of a good representation.”
- Learning curves for continual learning in neural networks: Self-knowledge transfer and forgetting (2021, background): “The current work and other theories in continual learning or transfer learning basically assume the same input distribution between different tasks (Lee et al., 2021; Tripuraneni et al., 2020).”
- Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution (2022, unknown): “Formally, the task diversity assumption in Tripuraneni et al. (2020) is sufficient (but not necessary) for obtaining a good B0.”
- A Scaling Law for Syn2real Transfer: How Much Is Your Pre-training Effective? (2021, methodology): “Among others, some recent studies [16, 42, 62] derived an error bound of a fine-tuning task in the multi-task scenario based on complexity analysis; the bound takes an additive form O ( An − 1 / 2 + Bs − 1 / 2 ) , where n and s are the data size of pre-training and fine-tuning, respectively, with…”
- Discriminability-Transferability Trade-Off: An Information-Theoretic Perspective (2022, background): “The transferability of deep representations has been studied from various perspectives [23, 36, 11, 43, 52, 24, 3, 7].”
- What Makes Multimodal Learning Better than Single (Provably) (2021, methodology): “Assumption 1 is a classical regularity condition for loss function in theoretical analysis [27, 41, 40].”
- In-N-Out: Pre-Training and Self-Training using Auxiliary Information for Out-of-Distribution Robustness (2020, methodology): “[46] study pre-training in a similar linear regression setup.”
- Meta-Adaptive Nonlinear Control: Theory and Algorithms (2021, background): “Task diversity has been shown to be important for representation learning [4, 33].”
- Provable and Efficient Continual Representation Learning (2022, methodology): “It is also in line with the intuitions from multitask representation learning (Maurer et al. (2016); Tripuraneni et al. (2020b)) and motivates the following CL principle supported by our experiments: First learn diverse and large-data tasks so that their representations help upcoming tasks.”
- Exploiting Shared Representations for Personalized Federated Learning (2021, methodology): “…of tasks is a classical approach in multi-task learning [Ando et al., 2005, Balcan et al., 2015, Baxter, 2000, Bengio et al., 2013, Bullins et al., 2019, Denevi et al., 2018, Kong et al., 2020, LeCun et al., 2015, Pontil and Maurer, 2013, Rish et al., 2008, Tripuraneni et al., 2020b].”