An Information-Maximization Approach to Blind Separation and Blind Deconvolution

By Anthony J. Bell, Terrence J. Sejnowski. 1995

Subfields: Natural Language Processing, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Optimization & Training

Citations in the corpus

Fast large-scale optimization by unifying stochastic gradient and quasi-Newton methods (2013, methodology): “We trained an Independent Components Analysis (ICA) (Bell & Sejnowski, 1995) We trained an Ising/Hopﬁeld model on MNIST digits, us-ing code from (Hillar et al., 2012).”
Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design (2019, methodology): “A ﬂow model f is constructed as an invertible transformation that maps observed data x to a standard Gaussian latent variable z = f ( x ) , as in nonlinear independent component analysis (Bell & Sejnowski, 1995; Hyv¨arinen et al., 2004; Hyv¨arinen & Pajunen, 1999).”
Density estimation using Real NVP (2016, background): “However, naive application of the change of variable formula produces models which are computationally expensive and poorly conditioned, and so large scale models of this type have not entered general use.”
On Mutual Information Maximization for Representation Learning (2019, background): “One solution is to maximize the entropy of the representation instead (Bell and Sejnowski, 1995).”
Learning deep representations by mutual information estimation and maximization (2018, methodology): “Some of the older methods, such as independent component analysis (ICA, Bell & Sejnowski, 1995) and self-organizing maps (Kohonen, 1998) generally cannot represent complex relationships in data.”
Learning Robust and Multilingual Speech Representations (2020, neutral): “learn representations that describe data by maximizing mutual information between data and representation variables, have been explored for a long time in unsupervised representation learning (Linsker, 1988; Bell and Sejnowski, 1995).”
Sparse coding with an overcomplete basis set: a strategy employed by V1? (1997, methodology): “Among these are the algorithms of Comon (1994), Amari, Cichocki, & Yang (1996), and Bell & Sejnowski (1995).”
Deep learning in neural networks: An overview (2014, background): “Many do this to uncover and disentangle hidden underlying sources of signals (e.g., Andrade, Chacon, Merelo, & Moran, 1993; Bell & Sejnowski, 1995; Belouchrani, Abed-Meraim, Cardoso, & Moulines, 1997; Cardoso, 1994; Comon, 1994; Hyvärinen, Karhunen, & Oja, 2001; Jutten & Herault, 1991; Karhunen &…”
Information Dropout: Learning Optimal Representations Through Noisy Computation (2016, background): “In this section, we outline strong connections between Information Dropout, Variational Autoencoders (VAEs) [5], and Independent Component Analysis (ICA) [22], [23].”
On Variational Bounds of Mutual Information (2019, background): “…x ∼ p ( x ) , the goal is to learn a stochastic representation of the data p θ ( y | x ) that has maximal MI with X subject to constraints on the mapping (e.g. Bell & Sejnowski, 1995; Krause et al., 2010; Hu et al., 2017; van den Oord et al., 2018; Hjelm et al., 2018; Alemi et al., 2017).”
High-Dimensional Probability Estimation with Deep Density Models (2013, methodology): “lt to ﬂexibly parameterize the space of such invertible maps, however, let alone ﬁnd a transformation that results in a tractable density on the representation space. Independent components analysis [Bell and Sejnowski, 1995] and the related idea of a density network [MacKay and Gibbs, 1997] are exa”
Formal Limitations on the Measurement of Mutual Information (2018, methodology): “Another related framework is INFOMAX (Linsker, 1988; Bell & Sejnowski, 1995; Hjelm et al., 2018).”
A Survey of Inductive Biases for Factorial Representation-Learning (2016, background): “Maximization of non-Gaussianity has a connection to information theory as well: it is equivalent to maximizing the mutual information between the observed data and the learned representation [9, 21], and forms the basis of a group of models called Independent Components Analysis or ICA .”
Maximum Entropy Generators for Energy-Based Models (2019, background): “With x = G ( z ) the function of interest and x discrete-valued: This idea of maximizing mutual information between inputs and outputs of an encoder to maximize the entropy of its output dates back at least to Linsker (1988); Bell & Sejnowski (1995) and their work on Infomax.”
A Unifying Information-Theoretic Framework for Independent Component Analysis (2000, methodology): “Intuitive explanations have been proposed by Bell and Sejnowski (1995), Lambert and Bell (1997), Girolami and Fyfe (1997c) and Lee, Bell and Orglmeister (1997).”
Motor patterns in human walking and running. (2006, methodology): “As before, we ordered the ICA and NMF components so that their waveforms matched the corresponding FA components (Fig.”
Dimensionality Reduction: A Comparative Review (2008, methodology): “Techniques for Independent Component Analysis [11] are not included in our review, because they were mainly designed for blind-source separation.”