TAFE-Net: Task-Aware Feature Embeddings for Low Shot Learning

By Xin Wang, Fisher Yu, Ruth Wang, Trevor Darrell, Joseph E. Gonzalez. 2019

106 citations recorded by the source corpus.

Learning good feature embeddings for images often requires substantial training data. As a consequence, in settings where training data is limited (e.g., few-shot and zero-shot learning), we are typically forced to use a general feature embedding across prediction tasks. Ideally, we would like to construct feature embeddings that are tuned for the given task and even input image. In this work, we propose Task Aware Feature Embedding Networks (TAFE-Nets) to learn how to adapt the image representation to a new task in a meta learning fashion. Our network is composed of a meta learner and a prediction network, where the meta learner generates parameters for the feature layers in the prediction network based on a task input so that the feature embedding can be accurately adjusted for that task. We show that TAFE-Net is highly effective in generalizing to new tasks or concepts and evaluate the TAFE-Net on a range of benchmarks in zero-shot and few-shot learning. Our model matches or exceeds the state-of-the-art on all tasks. In particular, our approach improves the prediction accuracy of unseen attribute-object pairs by 4 to 15 points on the challenging visual attribute-object composition task.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Meta & Continual Learning

Citations in the corpus

Dynamic Neural Networks: A Survey (2021, background): “In addition to computational efficiency, dynamic models have also shown promising results for improving data efficiency in the scenario of few-shot learning [17], [18].”
Frustratingly Simple Few-Shot Object Detection (2020, methodology): “In meta-learning approaches, in addition to the base object detection model that is either single-stage or two-stage, a meta-learner is introduced to acquire class-level meta knowledge and help the model generalize to novel classes through feature re-weighting, such as FSRW and Meta R-CNN, or class-”
Deep Learning on Small Datasets without Pre-Training using Cosine Loss (2019, background): “Another approach to fewshot learning is meta-learning: training a learner on large datasets to learn from small ones [22, 30, 48].”
A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities (2022, background): “The additional semantic information is aligned with visual features by embedding loss functions [152] to largely reduce the cost of knowledge transfer.”
A New Meta-Baseline for Few-Shot Learning (2020, background): “While more and more meta-learning approaches (Snell et al., 2017; Sung et al., 2018; Gidaris & Komodakis, 2018; Sun et al., 2019; Wang et al., 2019; Finn et al., 2017; Rusu et al., 2019; Lee et al., 2019) are proposed for few-shot learning, very few efforts (Gidaris & Komodakis, 2018; Chen et al.,…”
Stochastic Conditional Generative Networks with Basis Decomposition (2019, background): “The parameter prediction of neural network is intensively studied under the context of few shot learning [1, 19, 28], which aims to customize a network to a new task adaptively and efﬁciently in a data-driven way.”
Meta-Baseline: Exploring Simple Meta-Learning for Few-Shot Learning (2020, methodology): “Motivated by this idea, many recent works [26, 6, 25, 30, 5, 22, 11, 33] focus on improving the meta-learning structure, and few-shot learning itself has become a common testbed for evaluating meta-learning algorithms.”
Hyper-LifelongGAN: Scalable Lifelong Learning for Image Conditioned Generation (2021, methodology): “This idea has been applied to applications in different research fields such as few shot learning [27, 33], image segmentation [2, 36] and generative models [28, 18, 16], which is the focus of our paper.”
Multi-Task Reinforcement Learning with Soft Modularization (2020, background): “Our work is closely related to (Rosen-baum et al., 2017; 2019; Purushwalkam et al., 2019; Wang et al., 2019).”