Free Lunch for Few-shot Learning: Distribution Calibration

By Shuo Yang, Lu Liu, Min Xu. International Conference on Learning Representations, 2021

382 citations recorded by the source corpus.

Learning from a limited number of samples is challenging since the learned model can easily become overfitted based on the biased distribution formed by only a few training examples. In this paper, we calibrate the distribution of these few-sample classes by transferring statistics from the classes with sufficient examples, then an adequate number of examples can be sampled from the calibrated distribution to expand the inputs to the classifier. We assume every dimension in the feature representation follows a Gaussian distribution so that the mean and the variance of the distribution can borrow from that of similar classes whose statistics are better estimated with an adequate number of samples. Our method can be built on top of off-the-shelf pretrained feature extractors and classification models without extra parameters. We show that a simple logistic regression classifier trained using the features sampled from our calibrated distribution can outperform the state-of-the-art accuracy on two datasets (~5% improvement on miniImageNet compared to the next best). The visualization of these generated features demonstrates that our calibrated distribution is an accurate estimation.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/48c6d0d7e3fabf4692bd03fc8b7263e55ee1d584

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yu Qiao
2. Chunhua Shen
3. Hongsheng Li
4. Jie Tang
5. Mohit Bansal
6. Peng Gao
7. Guosheng Lin
8. Ming Ding

Citations in the corpus

• DeepEMD: Differentiable Earth Mover's Distance for Few-Shot Learning (2020, background): “classification problem, such as works based on graph theories [66], [67], [68], [69], reinforcement learning [70], differentiable SVM [71], generative models [72], [73], [74], [75], [76], [77], [78], [79], [80], transductive learning [81], [82], [83], [84], [85], recurrent models [86], [87], self-su”
• Prompt, Generate, Then Cache: Cascade of Foundation Models Makes Strong Few-Shot Learners (2023, background): “Many previous works have been proposed in this field to enhance model’s generalization capability by meta learning [20, 70], metric learning [73], and data augmentation [28, 72].”
• FewNLU: Benchmarking State-of-the-Art Methods for Few-Shot Natural Language Understanding (2021, methodology): “…al., 2018), prompting-based methods with discrete prompts (Schick and Schütze, 2021b; Gao et al., 2020) or continuous prompts (Liu et al., 2021b; Shin et al., 2020; Li and Liang, 2021; Lester et al., 2021), and methods that calibrate the output distribution (Yang et al., 2021; Zhao et al., 2021).”
• A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities (2022, methodology): “Recently, an assumption [12] is made that, when the base class and new class are semantically similar, the means and variances could be shared to a large extent.”
• Curriculum-Based Meta-learning (2021, neutral): “Additionally, there are also other methods that improve model generalization performance by generating extended episodes[1, 44, 49].”