Why Do Better Loss Functions Lead to Less Transferable Features?

By Simon Kornblith, Ting-Li Chen, Honglak Lee, Mohammad Norouzi. 2020

108 citations recorded by the source corpus.

Previous work has proposed many new loss functions and regularizers that improve test accuracy on image classification tasks. However, it is not clear whether these loss functions learn better representations for downstream tasks. This paper studies how the choice of training objective affects the transferability of the hidden representations of convolutional neural networks trained on ImageNet. We show that many objectives lead to statistically significant improvements in ImageNet accuracy over vanilla softmax cross-entropy, but the resulting fixed feature extractors transfer substantially worse to downstream tasks, and the choice of loss has little effect when networks are fully fine-tuned on the new tasks. Using centered kernel alignment to measure similarity between hidden representations of networks, we find that differences among loss functions are apparent only in the last few layers of the network. We delve deeper into representations of the penultimate layer, finding that different objectives and hyperparameter combinations lead to dramatically different levels of class separation. Representations with higher class separation obtain higher accuracy on the original task, but their features are less useful for downstream tasks. Our results suggest there exists a trade-off between learning invariant features for the original task and features relevant for transfer tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Renchunzi Xie
2. Bo An
3. Hongxin Wei
4. Lei Feng
5. Hao Cheng
6. Jiajun Liang
7. Osamu Yoshie
8. Quan Cui

In Computer Vision and Pattern Recognition

1. Renchunzi Xie
2. Bo An
3. Hongxin Wei
4. Lei Feng
5. Hao Cheng
6. Jiajun Liang
7. Osamu Yoshie
8. Quan Cui

Citations in the corpus

• Margin-Based Few-Shot Class-Incremental Learning with Class-Level Overfitting Mitigation (2022, background): “However, the model’s generalization capability on the downstream novel classes is much less explored and still needs to be improved [20, 14].”
• Mitigating Neural Network Overconfidence with Logit Normalization (2022, background): “A recent study (Kornblith et al., 2021) shows that several loss functions, including logit normalization and cosine softmax, lead to higher accuracy on ImageNet but degrade the performance of transfer tasks.”
• Discriminability-Transferability Trade-Off: An Information-Theoretic Perspective (2022, background): “The transferability of deep representations has been studied from various perspectives [23, 36, 11, 43, 52, 24, 3, 7].”