Few-Shot Learning via Saliency-Guided Hallucination of Samples

By Hongguang Zhang, Jing Zhang, Piotr Koniusz. 2019

216 citations recorded by the source corpus.

Learning new concepts from a few of samples is a standard challenge in computer vision. The main directions to improve the learning ability of few-shot training models include (i) a robust similarity learning and (ii) generating or hallucinating additional data from the limited existing samples. In this paper, we follow the latter direction and present a novel data hallucination model. Currently, most datapoint generators contain a specialized network (i.e., GAN) tasked with hallucinating new datapoints, thus requiring large numbers of annotated data for their training in the first place. In this paper, we propose a novel less-costly hallucination method for few-shot learning which utilizes saliency maps. To this end, we employ a saliency network to obtain the foregrounds and backgrounds of available image samples and feed the resulting maps into a two-stream network to hallucinate datapoints directly in the feature space from viable foreground-background combinations. To the best of our knowledge, we are the first to leverage saliency maps for such a task and we demonstrate their usefulness in hallucinating additional datapoints for few-shot learning. Our proposed network achieves the state of the art on publicly available datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Media Technology, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Media Technology

In Representation & Self-Supervised Learning

Citations in the corpus

Survey of Hallucination in Natural Language Generation (2022, background): “1The term “hallucination” first appeared in Computer Vision (CV) in Baker and Kanade [5] and carried more positive meanings, such as superresolution [5, 112], image inpainting [48], and image synthesizing [226].”
Incremental Few-Shot Object Detection (2020, neutral): “Note that there are many more recent FSL methods [6, 26, 7, 17, 47, 55, 50, 27, 39, 11, 16] that produce better performance on image classification tasks than MAML.”
Interventional Few-Shot Learning (2020, methodology): “These methods learn to generate additional support samples with image deformation [12, 74] or generative models [2, 75].”
Long-tailed Recognition by Routing Diverse Distribution-Aware Experts (2020, unknown): “A BSTRACT (b) Histograms of the largest softmax probability of the other classes (the hardest negative) per instance.”
Prior Guided Feature Enrichment Network for Few-Shot Segmentation (2020, unknown): “Data is essential to deep models; therefore, several methods improve performance by synthesizing more training samples [11], [42], [49].”
Leveraging the Feature Distribution in Transfer-based Few-Shot Learning (2020, methodology): “Other works [41, 4] utilize data augmentation techniques to artificially increase the size of the training datasets.”
Generalized Few-shot Semantic Segmentation (2020, background): “Therefore, synthesizing new training samples or features based on few labeled data is also a feasible solution for tackling the few-shot problem [52, 14, 42].”
Bowtie Networks: Generative Modeling for Joint Few-Shot Recognition and Novel-View Synthesis (2020, background): “, 2017), including the recent efforts on leveraging generative models (Li et al., 2015; Wang et al., 2018; Schwartz et al., 2018; Zhang et al., 2018; Tsutsui et al., 2019; Chen et al., 2019b; Li et al., 2019; Zhang et al., 2019; Sun et al., 2019).”
A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities (2022, methodology): “[50] further demonstrate the usefulness of phantom data generation for FSL and propose a low-cost automated data generation method that uses a direct foreground-background combination to generate feature space-level data for training.”
DeepEMD: Differentiable Earth Mover's Distance for Few-Shot Learning (2020, background): “classification problem, such as works based on graph theories [66], [67], [68], [69], reinforcement learning [70], differentiable SVM [71], generative models [72], [73], [74], [75], [76], [77], [78], [79], [80], transductive learning [81], [82], [83], [84], [85], recurrent models [86], [87], self-su”