Complex-Valued Autoencoders

By Pierre Baldi, Sholeh Forouzan, Zhiqin Lu. 2011

63 citations recorded by the source corpus.

Autoencoders are unsupervised machine learning circuits, with typically one hidden layer, whose learning goal is to minimize an average distortion measure between inputs and outputs. Linear autoencoders correspond to the special case where only linear transformations between visible and hidden variables are used. While linear autoencoders can be defined over any field, only real-valued linear autoencoders have been studied so far. Here we study complex-valued linear autoencoders where the components of the training vectors and adjustable matrices are defined over the complex field with the L2 norm. We provide simpler and more general proofs that unify the real-valued and complex-valued cases, showing that in both cases the landscape of the error function is invariant under certain groups of transformations. The landscape has no local minima, a family of global minima associated with Principal Component Analysis, and many families of saddle points associated with orthogonal projections onto sub-space spanned by sub-optimal subsets of eigenvectors of the covariance matrix. The theory yields several iterative, convergent, learning algorithms, a clear understanding of the generalization properties of the trained autoencoders, and can equally be applied to the hetero-associative case when external targets are provided. Partial results on deep architecture as well as the differential geometry of autoencoders are also presented. The general framework described here is useful to classify autoencoders and identify general properties that ought to be investigated for each class, illuminating some of the connections between autoencoders, unsupervised learning, clustering, Hebbian learning, and information theory.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Deep Learning without Poor Local Minima (2016, background): “This is indeed the expression of a global minimum (Baldi & Hornik, 1989; Baldi & Lu, 2012).”
Deep Neural Networks with Multi-Branch Architectures Are Less Non-Convex (2018, background): “Deep linear neural network has received signiﬁcant attention in recent years [51, 35, 67, 44, 8, 28, 31, 9] because of its simple formulation 3 and its connection to non-linear neural networks.”
Learning the Structural Vocabulary of a Network (2017, methodology): “Prior work has established a correspondence between PCA and autoencoders with linear hidden units (Baldi & Lu, 2012).”
An Empirical Analysis of Deep Network Loss Surfaces (2016, background): “Some have studied the critical points of the deep linear neural networks (Baldi, 1989; Baldi & Hornik, * Work done during an internship at Janelia Research Campus 1989; Baldi & Lu, 2012).”
Research on denoising sparse autoencoder (2016, background): “Autoencoder can adaptively extract images’ inherent features [3–7].”
Boolean autoencoders and hypercube clustering complexity (2012, background): “With the proper adjustments, the results above remain similar if a set of target output vectors y 1 , . . . , y m is provided, instead of x 1 , . . . , x m serving as the targets (see [3,4]).”
Learning in the Machine: Random Backpropagation and the Deep Learning Channel (2016, background): “By the results in [2,3], the solution must satisfy B A At = At .”
Deep Linear Networks with Arbitrary Loss: All Local Minima Are Global (2017, background): “For instance, both (Baldi & Hornik, 1989) and (Baldi & Lu, 2012) consider “deep” linear networks with two layers (i.e.”
Global optimization issues in deep network regression: an overview (2018, background): “More recently in [8] a partial extension to the case of deep linear autoencoder has been provided, namely that E (ω) has a block-wise convex structure in the sense that it is convex in each layer weights W k when the other weights W h ( h (cid:12)= k ) are ﬁxed.”