Loss Functions for Discriminative Training of Energy-Based Models

By Yann LeCun, Fu Jie Huang. 2005

163 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Scientific Benchmarks for Guiding Macromolecular Energy Function Improvement (2013, background): “Because of the vast size of conformation space, computing Z is often intractable; this is a common problem for energy-based loss functions (LeCun & Jie, 2005).”
Representation Matters: Offline Pretraining for Sequential Decision Making (2021, background): “Previous work has established connections between the use of contrastive loss and mutual information maximization (van den Oord et al., 2019) and energy-based models (LeCun and Huang, 2005).”
Structured Prediction Energy Networks (2015, background): “As LeCun & Huang (2005) discuss, the loss function for un-normalized models is unaffected by “low-energy areas that are never reached by the inference algorithm .””
A Primer on Neural Network Models for Natural Language Processing (2015, background): “For a detailed discussion on loss functions for neural networks see (LeCun, Chopra, Hadsell, Ranzato, & Huang, 2006; LeCun & Huang, 2005; Bengio et al., 2015).”
Learning a similarity metric discriminatively, with application to face verification (2005, background): “While probabilistic models assign a normalized probability to every possible configuration of the variables being modeled, energy-based models (EBM) assign an unnormalized energy to those configurations [18, 9].”
Learning optimized MAP estimates in continuously-valued MRF models (2009, methodology): “[16] and the Energy-Based Models [7], we learn the MRF parameters by defining a loss function that measures the similarity between the ground truth t and the minimum energy configuration of the MRF model.”
Learning to Combine Bottom-Up and Top-Down Segmentation (2006, unknown): “Then selectingα~λ∗, αν with α → ∞ will maximize the CRF likelihood, since: P (xt|It; α~λ∗, αν, ~ F ) = 1 (see [10]).”
Synergistic Face Detection and Pose Estimation with Energy-Based Models (2004, methodology): “1 Energy Minimization Framework We propose the following configuration of the Energy Based Model (LeCun and Huang, 2005; LeCun et al., 2006).”
Training Deep Neural Networks via Direct Loss Minimization (2015, background): “An alternative approach, frequently used in deep neural networks, is to train with a surrogate loss that can be easily optimized, e.g ., cross-entropy (LeCun & Huang, 2005; Bengio et al., 2015).”
Exploiting Inference for Approximate Parameter Learning in Discriminative Fields: An Empirical Study (2005, background): “Lecun and Huang [ 20 ] have described the sufficient conditions for the training of energybased (unnormalized) graphical models.”
Deep learning for side-channel analysis and introduction to ASCAD database (2019, background): “The cross-entropy [25,40] metric is a classical (and often by default) choice in classiﬁcation problems.”
Bilevel Optimization with Nonsmooth Lower Level Problems (2015, methodology): “[5], Hinton’s contrastive divergence method [6] and discriminative learning of graphical models [7, 8].”
A Neural Probabilistic Structured-Prediction Method for Transition-Based Natural Language Processing (2017, methodology): “As an alternative to maximize the likelihood on some observed data, contrastive learning (Hinton, 2002; LeCun & Huang, 2005; Liang & Jordan, 2008; Vickrey et al., 2010; Liu & Sun, 2014) is an approach that assigns higher probabilities to observed data and lower probabilities to noisy data.”
Training energy-based models for time-series imputation (2013, methodology): “It is also similar to the Recurrent Neural Network Restricted Boltzmann Machine (Boulanger-Lewandowski et al., 2012) that will be used as a baseline in our experiments.”
Toward automatic phenotyping of developing embryos from videos (2005, methodology): “The energy gets particularly large if 1(X), and f2(X) agree Fig.”
A Tutorial on Energy-Based Learning (2006, background): “Square-Square Loss: Unlike the hinge loss, the square-square loss treats the energy of the correct answer and the most offending answer separately [45, 30]: Lsq−sq(W,Y , X ) = E(W,Y , X ) + ( max(0,m− E(W, Ȳ , X )) 2 .”
Learning Speaker-Specific Characteristics With a Deep Neural Architecture (2011, background): “In general, (2) defines a multiobjective function and α is a constant used to trade off the contrastive loss in (3b) which hinders our architecture from learning speakerspecific characteristics against the reconstruction loss in (3a) that results in information loss.”