Few-shot learning of neural networks from scratch by pseudo example optimization

By Akisato Kimura, Zoubin Ghahramani, Koh Takeuchi, Tomoharu Iwata, Naonori Ueda. 2018

55 citations recorded by the source corpus.

In this paper, we propose a simple but effective method for training neural networks with a limited amount of training data. Our approach inherits the idea of knowledge distillation that transfers knowledge from a deep or wide reference model to a shallow or narrow target model. The proposed method employs this idea to mimic predictions of reference estimators that are more robust against overfitting than the network we want to train. Different from almost all the previous work for knowledge distillation that requires a large amount of labeled training data, the proposed method requires only a small amount of training data. Instead, we introduce pseudo training examples that are optimized as a part of model parameters. Experimental results for several benchmark datasets demonstrate that the proposed method outperformed all the other baselines, such as naive training of the target model and standard knowledge distillation.

Subfields: Electrical and Electronic Engineering, Mechanical Engineering, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Electrical and Electronic Engineering

In Mechanical Engineering

Dacheng Tao

In Meta & Continual Learning

Citations in the corpus

Zero-shot Knowledge Transfer via Adversarial Belief Matching (2019, methodology): “This is an important difference with competing models such as that of Kimura et al. (2018) where authors must include hand designed losses like carbon copy memory replay (which freezes some pseudo samples in time), or ﬁdelity (Dehghani et al., 2017). x adv ← x adv − ξ ∂ L CE ( y A , j ) ∂ x adv…”
Zero-Shot Knowledge Distillation in Deep Networks (2019, methodology): “Similar to MNIST, ZSKD outperforms the existing few data knowledge distillation approach (Kimura et al., 2018) by a large margin, and performs close to the classical knowledge distillation scenario (Hinton et al.”
Knowledge Distillation: A Survey (2020, methodology): “s. Similar to zero-shot learning, a knowledge distillation method with few-shot learning is designed by distilling knowledgefrom a teacher model with gaussian processes into a student neural network (Kimura et al., 2018). The teacher uses limited labelled data. 6.7 Quantized Distillation Network qua”
Well-Read Students Learn Better: The Impact of Student Initialization on Knowledge Distillation (2019, methodology): “Prior works have also noted the unavailability of in-domain large-scale transfer data and proposed the use of automatically generated pseudo-examples (Bucil ˘ a et al., 2006; Kimura et al., 2018).”
Knowledge Extraction with No Observable Data (2019, methodology): “[15] used knowledge distillation to effectively use unlabeled data for semi-supervised learning when a trained teacher network is given.”
Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, methodology): “[107] adopt the idea of inducing points [200] to generate pseudo training examples, which are then updated by applying adversarial examples [66], [207] and further optimized by an imitation loss.”
Distillation-Based Training for Multi-Exit Architectures (2019, methodology): “Since then, the technique has been found to benefit a wide array of applications, including transferring from one architecture to another [12], compression [17, 32], integration with first-order logic [18] or other prior knowledge [47], learning from noisy labels [26], defending against adversarial ”
Knowledge Distillation from Few Samples (2018, methodology): “Some recent works on knowledge distillation address this problem by constructing "pseudo" training data (Kimura et al., 2018; Lopes et al., 2017) with complicated heuristics and heavy engineering, which are still costly.”