Adaptive Online Learning Algorithms for Blind Separation: Maximum Entropy and Minimum Mutual Information

By Howard H. Yang, Shun-ichi Amari. 1997

There are two major approaches for blind separation: maximum entropy (ME) and minimum mutual information (MMI). Both can be implemented by the stochastic gradient descent method for obtaining the demixing matrix. The MI is the contrast function for blind separation; the entropy is not. To justify the ME, the relation between ME and MMI is first elucidated by calculating the first derivative of the entropy and proving that the mean subtraction is necessary in applying the ME and at the solution points determined by the MI, the ME will not update the demixing matrix in the directions of increasing the cross-talking. Second, the natural gradient instead of the ordinary gradient is introduced to obtain efficient algorithms, because the parameter space is a Riemannian space consisting of matrices. The mutual information is calculated by applying the Gram-Charlier expansion to approximate probability density functions of the outputs. Finally, we propose an efficient learning algorithm that incorporates with an adaptive method of estimating the unknown cumulants. It is shown by computer simulation that the convergence of the stochastic descent algorithms is improved by using the natural gradient and the adaptively estimated cumulants.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Disentangling by Factorising (2018, background): “This dates back to Yang & Amari (1997) who learn a linear map from representations to factors in the context of linear ICA, and quantify how close this map is to a permutation matrix.”
Kernel methods in machine learning (2007, neutral): “Also, the work of Chen and Bickel [28] and Yang and Amari [155] is of interest in this context.”
A Unifying Information-Theoretic Framework for Independent Component Analysis (2000, methodology): “73 and assuming unit variance for ui ( 2 u = 1), the form for the Bussgang nonlinear estimator must satisfy:f(si) = @p(si) @si p(si) (79) which is proportional to the derivative of the log-density of the true source distribution.”
Continuous latent variable models for dimensionality reduction and sequential data reconstruction (2001, methodology): “However, rescaling the gradient by a metric matrix AT A, we obtain a natural gradient algorithm (Yang and Amari, 1997; Amari and Cichoki, 1998): ∆A ∝ (I + g(u)u )A (2.”
Disentangling Disentanglement in Variational Autoencoders (2018, methodology): “The idea stems back to traditional methods such as ICA Hyvärinen and Oja (2000); Yang and Amari (1997) and conventional autoencoders Schmidhuber (1992), through to a range of modern approaches employing deep learning Achille and Soatto (2019) Of particular relevance to this work are approaches that…”
Source separation in post-nonlinear mixtures (1999, unknown): “As proposed by a few authors [8], [28], [38], [41], statistical independence can be measured using the Kullback–Liebler (KL) divergence between and”
Natural Conjugate Gradient in Variational Inference (2007, methodology): “It has previously been applied to derive efficient natural gradient learning rules for maximum likelihood algorithms to problems such as independent component analysis (ICA) ( Yang and Amari, 1997; Amari, 1998) and multilayer perceptron (MLP) networks (Amari, 1998) as well as to analyze the propert”
Linear and nonlinear ICA based on mutual information (2007, methodology): “Such matrix W can be find by minimizing some contrast functions such as the mutual information of the output[4] or maximizing some criteria such as entropy[5].”
A Framework for the Quantitative Evaluation of Disentangled Representations (2018, unknown): “Yang & Amari (1997, sec.”
Capturing inter-subject variability with group independent component analysis of fMRI data: A simulation study (2012, methodology): “Further information on the InfoMax algorithm and optimization with natural gradient descent can be found in Bell and Sejnowski (1995) and Yang and Amari (1997).”
Natural Gradient Works Eciently in Learning (methodology): “However, this is much better than the direct inversion of the original (n + 1)m-dimensional matrix of G. Yang and Amari (1997a) performed a preliminary study on the performance of the natural gradient learning algorithm for a simple multilayer perceptron.”
Natural Gradient Works Efficiently in Learning (1998, methodology): “Yang and Amari (1997) performed a preliminary study on the performance of the natural gradient learning algorithm for a simple multilayer perceptron.”