Adaptive Cross-Modal Few-Shot Learning

By Chen Xing, Negar Rostamzadeh, Boris N. Oreshkin, Pedro H. O. Pinheiro. 2019

339 citations recorded by the source corpus.

Metric-based meta-learning techniques have successfully been applied to few-shot classification problems. In this paper, we propose to leverage cross-modal information to enhance metric-based few-shot learning methods. Visual and semantic feature spaces have different structures by definition. For certain concepts, visual features might be richer and more discriminative than text ones. While for others, the inverse might be true. Moreover, when the support from visual information is limited in image classification, semantic representations (learned from unsupervised text corpora) can provide strong prior knowledge and context to help learning. Based on these two intuitions, we propose a mechanism that can adaptively combine information from both modalities according to new image categories to be learned. Through a series of experiments, we show that by this adaptive combination of the two modalities, our model outperforms current uni-modality few-shot learning methods and modality-alignment methods by a large margin on all benchmarks and few-shot scenarios tested. Experiments also show that our model can effectively adjust its focus on the two modalities. The improvement in performance is particularly large when the number of shots is very small.

Subfields: Artificial Intelligence, Cancer Research, Computer Vision and Pattern Recognition, Multimodal Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cancer Research

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Meta & Continual Learning

Citations in the corpus

Baby steps towards few-shot learning with multiple semantics (2019, methodology): “Our implementation is built on top and extends the code kindly provided by the authors of [56], also keeping their hyper-parameter setting (such as learning rate schedules, etc).”
Boosting Few-Shot Learning With Adaptive Margin Loss (2020, methodology): “Meta-learning [7, 27, 31, 34, 35] is a common approach for the FSL.”
DeepEMD: Differentiable Earth Mover's Distance for Few-Shot Learning (2020, neutral): “ods [1], [2], [3], [4], [5], [6], [59], [60], [61], [62], [63], [64],”
Regression Networks for Meta-Learning Few-Shot Classification (2019, methodology): “We decide to compare with these methods in particular, because they serve as the basis of many state-of-the-art few-shot classiﬁcation algorithms (Oreshkin et al., 2018; Xing et al., 2019), and our method is easily interchanged with them.”
Contextualizing Multiple Tasks via Learning to Decompose (2021, methodology): “[38, 54, 68, 58, 59] adapts the feature embeddings or the model initialization given a few-shot support set, [48, 57] compose instance representations with the supervision of attributes, and [4] utilizes the ﬁne-grained visual concepts.”
SIN: Semantic Inference Network for Few-Shot Streaming Label Learning (2022, methodology): “Adaptive modality mixture mechanism (AM3) [16] introduces semantic representation to adapt the prototype and achieves state-of-the-art performance on few-shot learning.”
Improved Few-Shot Visual Classification (2019, neutral): “The prototypes are often deﬁned in feature or semantic space ( e.g . word2vec [39]).”
CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understanding (2022, methodology): “Multi-modal setting has been adopted in various 2D vision tasks such as object detection [24], few-shot image classification [2,60] and visual question and answering [22, 56].”
Aligning Visual Prototypes with BERT Embeddings for Few-Shot Learning (2021, methodology): “65 AM3(ProtoNet++) [53] ResNet-12 Metric 65.”