Hybrid Discriminative-Generative Training via Contrastive Learning

By Hao Liu, Pieter Abbeel. 2020

43 citations recorded by the source corpus.

Contrastive learning and supervised learning have both seen significant progress and success. However, thus far they have largely been treated as two separate objectives, brought together only by having a shared neural network. In this paper we show that through the perspective of hybrid discriminative-generative training of energy-based models we can make a direct connection between contrastive learning and supervised learning. Beyond presenting this unified view, we show our specific choice of approximation of the energy-based loss outperforms the existing practice in terms of classification accuracy of WideResNet on CIFAR-10 and CIFAR-100. It also leads to improved performance on robustness, out-of-distribution detection, and calibration.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Arash Vahdat
2. Karsten Kreis
3. Jan Kautz
4. Jingfei Du
5. Ves Stoyanov
6. Zhisheng Xiao
7. Beliz Gunel
8. Jinwoo Shin

In Computer Vision and Pattern Recognition

1. Arash Vahdat
2. Karsten Kreis
3. Jan Kautz
4. Luc Van Gool
5. Jinwoo Shin
6. Ming-Hsuan Yang
7. Dacheng Tao
8. Zhisheng Xiao

In Representation & Self-Supervised Learning

1. Jingfei Du
2. Zhisheng Xiao
3. Ves Stoyanov
4. Jinwoo Shin
5. Jihoon Tack
6. Jan Kautz
7. Alexis Conneau
8. Karsten Kreis

Citations in the corpus

• VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models (2020, background): “Joint EBMs (Grathwohl et al., 2020a; Liu & Abbeel, 2020) model the joint distribution of data and labels.”
• Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning (2020, background): “Liu & Abbeel (2020) propose a hybrid discriminative-generative training of energy-based models where they approximate the generative term with a contrastive loss using large batch sizes and show improved classification accuracy of WideResNet-28-10 (Zagoruyko & Komodakis, 2016) on CIFAR-10 and…”
• Your Diffusion Model is Secretly a Zero-Shot Classifier (2023, unknown): “Moreover, such models [27, 50, 37] have been shown to generalize better, be more robust, and be better calibrated .”
• CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances (2020, background): “[68] and Liu and Abbeel [40] report that contrastive learning also improves the OOD detection performance of classifiers [39, 38, 25].”
• Consistent Representation Learning for Continual Relation Extraction (2022, background): “Liu and Abbeel (2020) proposed a hybrid discriminant-generative training method based on an energy model.”