Maximum Entropy Discrimination

By Tommi S. Jaakkola, Marina Meila, Tony Jebara. 1999

We present a general framework for discriminative estimation based on the maximum entropy principle and its extensions. All calculations involve distributions over structures and/or parameters rather than specific settings and reduce to relative entropy projections. This holds even when the data is not separable within the chosen parametric class, in the context of anomaly detection rather than classification, or when the labels in the training set are uncertain or incomplete. Support vector machines are naturally subsumed under this class and we provide several extensions. We are also able to estimate exactly and efficiently discriminative distributions over tree structures of class-conditional models within this framework. Preliminary experimental results are indicative of the potential in these techniques.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Variational Autoencoders for Collaborative Filtering (2018, neutral): “6 resembles maximum-entropy discrimination [20].”
Multi-view learning overview: Recent progress and new challenges (2017, background): “Margin-consistency style algorithms model the margin variables from multiple views to be consistent in the framework of MED [14].”
A Probabilistic Framework for the Hierarchic Organisation and Classification of Document Collections (2002, background): “It can also be derived from more general maximum entropy discrimination framework (Jaakkola et al., 1999, 2000).”
Multi-task feature and kernel selection for SVMs (2004, methodology): “The maximum entropy discrimination MED formalism introduced in (Jaakkola et al., 1999) is a flexible generalization of support vector machines.”
Infinite Latent SVM for Classification and Multi-task Learning (2011, background): “Recent attempts toward learning a posterior distribution of model parameters include the “learning from measurements” [19], maximum entropy discrimination [15] and MedLDA [31].”
PAC-Bayesian Generalisation Error Bounds for Gaussian Process Classification (2003, unknown): “Jaakkola et al. (2000) give some examples of such priors which encourage large margins and lead to an objective function log Z(λ) which resembles very much the dual objective of soft margin SV classification.”
Vicinal Risk Minimization (2000, result): “Recent results ( Jaakkola, Meila and Jebara, 2000 ) even define generative models that contain SVM as special cases.”
Discriminative Clustering by Regularized Information Maximization (2010, methodology): “[7] also use relative entropy as a means of enforcing prior beliefs, although not with respect to class distributions in multi-class classification problems.”
Comparison of machine learning and traditional classifiers in glaucoma diagnosis (2002, background): “Recently, there has been growing interest in combining the generative and discriminative approaches to train classifiers [63]–[65].”
Machine Learning Paradigms for Speech Recognition: An Overview (2013, background): “Similarly, Maximum entropy discrimination [10] seeks that minimizes under the constraints that .”
On Bayesian bounds (2006, methodology): “Several learning algorithms—such as the entropy projection viewpoint of boosting (Kivinen & Warmuth, 1999) as well as direct maximum entropy discrimination in the batch (Jaakkola et al., 1998) and online settings (Long & Wu, 2004)—follow the maximum entropy “principle.””
Looking at People: Sensing for Ubiquitous and Wearable Computing (2000, methodology): “SVMs are very closely related to Bayesian methods and Maximum Entropy methods for discriminative learning [16].”
Hyperparameter Learning for Graph Based Semi-supervised Learning Algorithms (2006, unknown): “(Jaakkola et al., 1999) proposed a maximum entropy discrimination model based on maximum margin.”
Introduction to Semi-Supervised Learning (2009, methodology): “It can be implemented in Gaussian Processes with the null category noise model [106, 40], as information regularization [169, 44, 45], maximum entropy discrimination approach [87], or entropy minimization [79, 108, 122].”
Infinite Markov-Switching Maximum Entropy Discrimination Machines (2013, methodology): “In addition, MED allows for incorporating latent variables in the derived models (Lewis et al., 2006), which comprise a key-tool in machine learning, as well as for performing structured output prediction (Zhu & Xing, 2009).”
Probability Product Kernels (2004, background): “Examples of such approaches include conditional learning (Bengio and Frasconi, 1996) or large margin generative modeling (Jaakkola et al., 1999).”