Empirical Bayes Transductive Meta-Learning with Synthetic Gradients

By Shell Xu Hu, Pablo G. Moreno, Yanghua Xiao, Xin Shen, Guillaume Obozinski, Neil D. Lawrence, Andreas C. Damianou. 2020

140 citations recorded by the source corpus.

We propose a meta-learning approach that learns from multiple tasks in a transductive setting, by leveraging unlabeled information in the query set to learn a more powerful meta-model. To develop our framework we revisit the empirical Bayes formulation for multi-task learning. The evidence lower bound of the marginal log-likelihood of empirical Bayes decomposes as a sum of local KL divergences between the variational posterior and the true posterior of each task. We derive a novel amortized variational inference that couples all the variational posteriors into a meta-model, which consists of a synthetic gradient network and an initialization network. The combination of local KL divergences and synthetic gradient network allows for backpropagating information from unlabeled data, thereby enabling transduction. Our results on the Mini-ImageNet and CIFAR-FS benchmarks for episodic few-shot classification significantly outperform previous state-of-the-art methods.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Few-Shot and Continual Learning with Attentive Independent Mechanisms (2021, methodology): “The algorithms shown are applicable for both few-shot and continual learning with the distinction between both highlighted with different colors — few-shot learning using SIB in green and continual learning using OML and ANML in blue.”
Interventional Few-Shot Learning (2020, methodology): “We chose linear classiﬁer as it outperforms cosine and k -NN in cross-domain setting and compared with transductive method — SIB.”
A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities (2022, unknown): “54% SIB [235] ICLR’20 Meta WRN ✓ ✓ ✓ 70.”
Adaptive Risk Minimization: A Meta-Learning Approach for Tackling Group Shift (2020, background): “, 2019) or the transductive learning setting (Liu et al., 2019; Antoniou & Storkey, 2019; Hu et al., 2020).”
Few-Shot Object Detection and Viewpoint Estimation for Objects in the Wild (2020, background): “One popular solution to this problem is meta-learning [25, 4, 2, 58, 56, 48, 21, 41, 14, 27, 22], where a meta-learner is designed to parameterize the optimization algorithm or predict the network parameters by ”learning to learn”.”
Meta-Learning with Shared Amortized Variational Inference (2020, unknown): “On the 5-shot task, higher accuracy is reported by Dhillon et al. (2020) and Gidaris et al. (2019), while transductive SIB (Hu et al., 2020) is comparable to SAMOVAR.”
DeepEMD: Differentiable Earth Mover's Distance for Few-Shot Learning (2020, background): “classification problem, such as works based on graph theories [66], [67], [68], [69], reinforcement learning [70], differentiable SVM [71], generative models [72], [73], [74], [75], [76], [77], [78], [79], [80], transductive learning [81], [82], [83], [84], [85], recurrent models [86], [87], self-su”