Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One

By Will Grathwohl, Kuan-Chieh Jackson Wang, Jörn-Henrik Jacobsen, David Kristjanson Duvenaud, Mohammad Norouzi, Kevin Swersky. 2019

664 citations recorded by the source corpus.

We propose to reinterpret a standard discriminative classifier of p(y|x) as an energy based model for the joint distribution p(x,y). In this setting, the standard class probabilities can be easily computed as well as unnormalized values of p(x) and p(x|y). Within this framework, standard discriminative architectures may beused and the model can also be trained on unlabeled data. We demonstrate that energy based training of the joint distribution improves calibration, robustness, andout-of-distribution detection while also enabling our models to generate samplesrivaling the quality of recent GAN approaches. We improve upon recently proposed techniques for scaling up the training of energy based models and presentan approach which adds little overhead compared to standard classification training. Our approach is the first to achieve performance rivaling the state-of-the-artin both generative and discriminative learning within one hybrid model.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Probabilistic & Bayesian Methods

Citations in the corpus

Denoising Diffusion Probabilistic Models (2020, neutral): “9 JEM [17] 8.”
Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Sampling (2020, background): “Other work has built new connections between energy based models and classiﬁers [37].”
Flow Contrastive Estimation of Energy-Based Models (2019, background): “On the other end of the spectrum of deep generative models lies the family of energy-based models (EBMs) [42, 54, 29, 77, 73, 13, 38, 55, 9, 11, 16, 18, 6].”
Diffusion Models Beat GANs on Image Synthesis (2021, methodology): “[17] train a model which is jointly a classifier and an energy-based model.”
VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models (2020, background): “Joint EBMs (Grathwohl et al., 2020a; Liu & Abbeel, 2020) model the joint distribution of data and labels.”
Energy-based Latent Aligner for Incremental Learning (2022, background): “Joint Energy-based Model (JEM) [14] shows that any classiﬁer can be reinterpreted as a generative model that can model the joint likelihood of labels and data.”
Supermasks in Superposition (2020, methodology): “As future work, we hope to explore automatic task inference with more calibrated models [14], as well as circumventing calibration challenges by using optimization objectives such as self-supervision [16] and energy based models [13].”
Learning Energy-Based Models by Diffusion Recovery Likelihood (2020, background): “EBMs [27, 33, 21, 47, 10, 43, 6, 7, 25, 35, 5, 11, 4, 8, 2, 12] are an appealing class of probabilistic models, which can be viewed as generative versions of discriminators [18, 26, 28, 12], yet can be learned from unlabeled data.”
Energy-based Out-of-distribution Detection (2020, methodology): “This is in contrast with JEM [11], which derives the likelihood score log p ( x ) from a generative modeling perspective.”
Variational (Gradient) Estimate of the Score Function in Energy-based Latent Variable Models (2020, neutral): “1 (a), we numerically validate Theorem 1,2 in Gaussian restricted Boltzmann machines (GRBMs) (Hinton & Salakhutdinov, 2006).”
Discriminator Contrastive Divergence: Semi-Amortized Generative Modeling by Exploring Energy of the Discriminator (2020, background): “With a similar spirit, [11] also illustrates the potential connection 274 between neural classifier and energy-based model in supervised and semi-supervised scenarios.”
A Unifying Review of Deep and Shallow Anomaly Detection (2020, methodology): “These models are typically trained via gradient descent, approximating the log-likelihood gradient ∇θ log pθ(x) via MCMC [265] or SGLD [266], [267].”
See through Gradients: Image Batch Recovery via GradInversion (2021, methodology): “To further encourage exploration and diversity, we add pixel-wise random Gaussian noise in each update, inspired by the Langevin updates in energy-based models [10, 12, 15].”
Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, background): “Energy-based Models (EBMs) [26, 48, 58, 64, 67, 68, 75, 78, 79, 120, 129, 132, 165, 170, 182, 196, 254, 281] can be viewed as one generative version of discriminators [79, 104, 131, 134], while can be learned from unlabeled input data.”
Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models (2021, methodology): “By marginalising out y, these logits can be used to define an energy model that can be simultaneously trained as both a generative and classification model [64],”
Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (2023, background): “Furthermore, it can be effectively applied to universal generative models, even those without explicit likelihood (Goodfellow et al., 2014; Grathwohl et al., 2019; Du & Mordatch, 2019).”
Tradeoffs in Data Augmentation: An Empirical Study (2021, methodology): “We consider the mean log likelihood of augmented test images (Grathwohl et al., 2019), and the Watanabe–Akaike information criterion (WAIC) (Watanabe, 2010).”