Information Theoretic Learning

By Deniz Erdoğmuş, José Carlos Príncipe. Encyclopedia of Artificial Intelligence, 2005

851 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Signal Processing

View Paper: https://www.semanticscholar.org/paper/0c9ade34d579e190a20fd4e31ad9fe8364515357

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

Robust support vector machines based on the rescaled hinge loss function (2017, background): “Correntropy is an information theoretic metric [20] and has been commonly used in robust learning [21].”
Feature Extraction by Non-Parametric Mutual Information Maximization (2003, methodology): “This resulted in no loss or minimal loss of accuracy compared to full supervised training of all network parameters using MMI.”
Prototype-based Neural Network Layers: Incorporating Vector Quantization (2018, background): “…is mathematically supported by strong theoretical results regarding the convergence of prototype-based methods [70, 71], hypothesis margin optimization by LVQ networks [60] and other aspects like information theoretic properties, Bayesian decision theory for LVQ networks and more [72, 73, 74, 48].”
Robust Principal Component Analysis Based on Maximum Correntropy Criterion (2011, methodology): “It is derived from the generalized correlation function of random processes and is directly related to the information potential (IP) of Renyi’s quadratic entropy [16] in which Parzen windowing method is used to estimate the data’s probability distribution [22].”
Generalized correlation function: definition, properties, and application to blind equalization (2006, background): “Likewise, advances in information theoretic learning (ITL), have brought out a number of applications, where entropy and divergence employ Parzen’s nonparametric estimation [8]–[11].”
Normalized Mutual Information Feature Selection (2009, neutral): “In future research, it would be of interest to use quadratic MI [47] for estimating MI between high-dimensional vectors, and to use the concept of Markov blanket for finding features that are weakly relevant but nonredundant.”
Correntropy: Properties and Applications in Non-Gaussian Signal Processing (2007, methodology): “Finally, Section VI summarizes the main conclusions and future lines of research.”
Information, Divergence and Risk for Binary Experiments (2009, background): “A large collection is due to Rachev (1991). A recent survey on relationships (inequalities and some representations) has been given by Gibbs and Su (2002).”
On Loss Functions for Deep Neural Networks in Classification (2017, background): “At the same time this information theoretic measure is very rarely used in DL community, and rather exploited in shallow learning (for both classification [10] and clustering [13]).”
The design and analysis of information processing systems ] From Linear Adaptive Filtering to Nonlinear Information Processing (2009, methodology): “Consider the quadratic information potential [45], [17] Note that the information potential of the samples in the training , now called information particles {x 1 , . . . , x N } is the average of the information potentials experienced by individual particles, denoted by V 2 (x j ).”
Causality detection based on information-theoretic approaches in time series analysis (2007, methodology): “For example, classiﬁcation and pattern recognition literature uses it them for feature extraction, i.e. Torkkola [233] or for principal or independent component analysis, i.e. Xu et al. [254,184,185].”
An introductory survey of probability density function control (2019, methodology): “Since MEE criterion ensures that the estimation error has small uncertainty, it was used for supervised training of non-linear stochastic system by Erdogmus and Principe (2002); Principe (2010).”