Improved Contrastive Divergence Training of Energy Based Models

By Yilun Du, Shuang Li, Josh Tenenbaum, Igor Mordatch. 2020

180 citations recorded by the source corpus.

We propose several different techniques to improve contrastive divergence training of energy-based models (EBMs). We first show that a gradient term neglected in the popular contrastive divergence formulation is both tractable to estimate and is important to avoid training instabilities in previous models. We further highlight how data augmentation, multi-scale processing, and reservoir sampling can be used to improve model robustness and generation quality. Thirdly, we empirically evaluate stability of model architectures and show improved performance on a host of benchmarks and use cases, such as image generation, OOD detection, and compositional generation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sergey Levine
2. Josh Tenenbaum
3. Arash Vahdat
4. Chelsea Finn
5. Pieter Abbeel
6. Igor Mordatch
7. Yilun Du
8. Anima Anandkumar

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Xiaogang Wang
3. Jitendra Malik
4. Trevor Darrell
5. Ming-Hsuan Yang
6. Dacheng Tao
7. Andrew Zisserman
8. Antonio Torralba

In Statistical and Nonlinear Physics

1. Arash Vahdat
2. Yilun Du
3. Yang Song
4. Igor Mordatch
5. Josh Tenenbaum
6. Diederik P. Kingma
7. Shuang Li
8. Anima Anandkumar

Citations in the corpus

• Implicit Behavioral Cloning (2021, background): “In applications, 222 energy based models have recently shown state-of-the-art results across a number of domains, including 223 various computer vision tasks [35, 36], as well as generative modeling tasks such as image and text 224 generation [12, 37, 38].”
• Compositional Visual Generation with Composable Diffusion Models (2022, methodology): “Energy-Based Models (EBMs) [7, 9, 12, 30] are a class of generative models where the data distribution is modeled using an unnormalized probability density.”
• Diffusion policy: Visuomotor policy learning via action diffusion (2023, unknown): “In practice, the inaccuracy of negative sampling is known to cause training instability for EBMs (Du et al., 2020; Ta et al., 2022).”
• Compositional Visual Generation with Energy Based Models (2020, methodology): “To enable high resolution image generation, we further apply techniques for improving EBM training described in [Du et al., 2020] on CelebA images.”
• How to Train Your Energy-Based Models (2021, background): “For example, one line of work proposes unbiased estimators of the gradient through coupled MCMC (Jacob et al., 2017; Qiu et al., 2019); and Du et al. (2020) propose to reduce the bias by differentiating through the MCMC sampling algorithm and estimating an entropy correction term.”
• Controllable and Compositional Generation with Latent-Space Energy-Based Models (2021, background): “However, existing approaches train EBMs directly in the pixel space, making them slow to sample and difficult to generate high-resolution images [13].”
• Pre-training Molecular Graph Representation with 3D Geometry (2021, background): “Recently, there have been quite a lot progress along this direction [17, 29, 69, 70].”