Adapted Deep Embeddings: A Synthesis of Methods for k-Shot Inductive Transfer Learning

By Tyler R. Scott, Karl Ridgeway, Michael C. Mozer. 2018

90 citations recorded by the source corpus.

The focus in machine learning has branched beyond training classifiers on a single task to investigating how previously acquired knowledge in a source domain can be leveraged to facilitate learning in a related target domain, known as inductive transfer learning. Three active lines of research have independently explored transfer learning using neural networks. In weight transfer, a model trained on the source domain is used as an initialization point for a network to be trained on the target domain. In deep metric learning, the source domain is used to construct an embedding that captures class structure in both the source and target domains. In few-shot learning, the focus is on generalizing well in the target domain based on a limited number of labeled examples. We compare state-of-the-art methods from these three paradigms and also explore hybrid adapted-embedding methods that use limited target-domain data to fine tune embeddings constructed from source-domain data. We conduct a systematic comparison of methods in a variety of domains, varying the number of labeled instances available in the target domain (k), as well as the number of target-domain classes. We reach three principal conclusions: (1) Deep embeddings are far superior, compared to weight transfer, as a starting point for inter-domain transfer or model re-use (2) Our hybrid methods robustly outperform every few-shot learning and every deep metric learning method previously proposed, with a mean error reduction of 34% over state-of-the-art. (3) Among loss functions for discovering embeddings, the histogram loss (Ustinova & Lempitsky, 2016) is most robust. We hope our results will motivate a unification of research in weight transfer, deep metric learning, and few-shot learning.

Subfields: Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

Citations in the corpus

Do Better ImageNet Models Transfer Better? (2018, background): “Common few-shot learning methods do not seem to outperform classifiers trained on fixed features when domain shift is present [8], but it may be possible to obtain better results with specialized methods [18] or by combining few-shot learning methods with fine-tuning [60].”
Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need? (2020, background): “Each task consists of limited training data, which requires fast adaptability [45] of the learner (e.”
Semantic Drift Compensation for Class-Incremental Learning (2020, background): “Furthermore, deep embed-dings were reported to be superior to classiﬁcation networks for transfer learning [35] and preliminary results suggest that they might be less prone to catastrophic forgetting [47].”
Meta-Transfer Learning for Few-Shot Learning (2018, methodology): “Some few-shot learning methods have been proposed to use pre-trained weights as initialization [20, 29, 37, 44, 40].”
Learning Adaptive Classifiers Synthesis for Generalized Few-Shot Learning (2019, methodology): “g objective that transfer the task-level inductive bias from SEEN classes to the new task composed by UNSEEN classes. For example, one line of works use meta-learned discriminative feature embeddings [16, 20, 25, 27, 29, 34, 39] together with non-parametric nearest neighbor classiﬁers, to 2 recogniz”
Meta-Transfer Learning Through Hard Tasks (2019, methodology): “Some few-shot learning methods have been proposed to utilize pre-trained DNNs [25], [44], [56]–[58].”
Few-Shot Learning via Embedding Adaptation With Set-to-Set Functions (2018, background): “Another line of approach has focused on learning generalizable instance embeddings [1, 4, 5, 13, 17, 26, 37, 40, 43] and uses those embeddings on simple classifiers such as nearest neighbor rules.”
Small Sample Learning in Big Data Era (2018, unknown): “Furthermore, a hybrid approach was proposed in (Scott et al., 2018) to combine deep embedding losses for training (metric learning) on the source domain with weight adaptation (domain adaptation) on the target domain for k-shot learning.”
Contextualizing Multiple Tasks via Learning to Decompose (2021, methodology): “[38, 54, 68, 58, 59] adapts the feature embeddings or the model initialization given a few-shot support set, [48, 57] compose instance representations with the supervision of attributes, and [4] utilizes the ﬁne-grained visual concepts.”
Few-Shot Learning With a Strong Teacher (2021, methodology): “The resulting features or metric are used in a linear or nearest-neighbor classifier that is trained on the examples of the novel classes [47], [48], [49], [11], [9].”
DeepEMD: Differentiable Earth Mover's Distance for Few-Shot Learning (2020, unknown): “, [26], [27], [28], [29], [30], [31], [32], [33], [34], [35], [36], [37], [38], [39], [40], [41], [42], [43], [44], [45], [46], [47], [48], [49], [50], [51], [52], [53], target at effectively adapting model parameters to new tasks in the low-shot regime.”
Learning Classifier Synthesis for Generalized Few-Shot Learning (2019, methodology): “g objective that transfer the task-level inductive bias from SEEN classes to the new task composed by UNSEEN classes. For example, one line of works use meta-learned discriminative feature embeddings [16, 20, 25, 27, 29, 34, 39] together with non-parametric nearest neighbor classiﬁers, to 2 recogniz”