CrossTransformers: spatially-aware few-shot transfer

By Carl Doersch, Ankush Gupta, Andrew Zisserman. 2020

412 citations recorded by the source corpus.

Given new tasks with very little data$-$such as new classes in a classification problem or a domain shift in the input$-$performance of modern vision systems degrades remarkably quickly. In this work, we illustrate how the neural network representations which underpin modern vision systems are subject to supervision collapse, whereby they lose any information that is not necessary for performing the training task, including information that may be necessary for transfer to new tasks or domains. We then propose two methods to mitigate this problem. First, we employ self-supervised learning to encourage general-purpose features that transfer better. Second, we propose a novel Transformer based neural network architecture called CrossTransformers, which can take a small number of labeled images and an unlabeled query, find coarse spatial correspondence between the query and the labeled images, and then infer class membership by computing distances between spatially-corresponding features. The result is a classifier that is more robust to task and domain shift, which we demonstrate via state-of-the-art performance on Meta-Dataset, a recent dataset for evaluating transfer from ImageNet to many other vision datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Media Technology, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Media Technology

In Meta & Continual Learning

Citations in the corpus

Flamingo: a Visual Language Model for Few-Shot Learning (2022, methodology): “[24] Carl Doersch, Ankush Gupta, and Andrew Zisserman.”
Visual Prompt Tuning (2022, background): “The triumph of the Transformer architecture also extends to various computer vision tasks, including image classification [16,48], object detection [7,45], semantic segmentation [66,83], panoptic segmentation [73], video understanding [21] and fewshot learning [15], surpassing previous state-of-the-”
Representational Continuity for Unsupervised Continual Learning (2021, background): “We conjecture that this is due to their characteristic ability to learn general-purpose features (Doersch et al., 2020), which makes them transfer better and comparatively more robust to catastrophic forgetting.”
Transformers in Vision: A Survey (2021, methodology): “(CT: Cross Transformers, AP: Average Precision, mAP: mean AP, IoU: Intersection over Union, FID: Fréchet inception distance, MPJPE: Mean Per Joint Position Error, MPVE: Mean Per Vertex Error).”
Segmenter: Transformer for Semantic Segmentation (2021, methodology): “The success of transformers in NLP has inspired several methods in computer vision combining CNNs with forms of self-attention to address object detection [7], semantic segmentation [53], panoptic segmentation [51], video processing [54] and few-shot classiﬁcation [18].”
Rethinking the Representational Continuity: Towards Unsupervised Continual Learning (2021, background): “We conjecture that this is due to their characteristic ability to learn general-purpose features (Doersch et al., 2020), which makes them transfer better and comparatively more robust to catastrophic forgetting.”
A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities (2022, background): “Recently, many works [96, 97] explore useful tricks, including self-supervised learning [96],”
Are Large-scale Datasets Necessary for Self-Supervised Pre-training? (2021, neutral): “[14], is the so-called supervision collapse .”
Keeping Your Eye on the Ball: Trajectory Attention in Video Transformers (2021, background): “Transformers have also been used in a variety of vision image tasks, such as image representation learning [12, 82, 18, 59], image generation [51], object detection [48, 10], video question-answering [35], few-shot learning [19], and image–text [49, 63, 67, 44, 68], video-text [65, 64, 89, 25, 53, 2”
Universal Representation Learning from Multiple Domains for Few-shot Classification (2021, methodology): “We follow the procedure in [12], sample a varying number of ways in Meta-Dataset as in the standard setting but a fixed number of”
Fine-Grained Few-Shot Classification with Feature Map Reconstruction Networks (2020, methodology): “We use Laux as the auxiliary loss in our subspace network implementation [19], and it replaces the SimCLR episodes in our CrossTransformer implementation [5].”
Improving Task Adaptation for Cross-domain Few-shot Learning (2021, methodology): “After evaluating our method over a broad range of varying shots ( e.g . up to 100 shots), we follow [10, 22] to further analyze our method in 5-shot setting of varying number of categories.”
Novel Visual Category Discovery with Dual Ranking Statistics and Mutual Knowledge Distillation (2021, background): “Part-level features have been shown to be effective for tasks involving image verification such as few-shot learning [10, 55, 12] and image retrieval [7, 34, 5, 2, 37, 42, 45].”
Why Do Better Loss Functions Lead to Less Transferable Features? (2020, methodology): “[22] motivate their few-shot learning method as a way to mitigate “supervision collapse,” which they do not directly quantify, but is closely related to our notion of class separation.”