Deep Neural Networks with Random Gaussian Weights: A Universal Classification Strategy?

By Raja Giryes, Guillermo Sapiro, A. Bronstein. 2015

191 citations recorded by the source corpus.

Three important properties of a classification machinery are i) the system preserves the core information of the input data; ii) the training examples convey information about unseen data; and iii) the system is able to treat differently points from different classes. In this paper, we show that these fundamental properties are satisfied by the architecture of deep neural networks. We formally prove that these networks with random Gaussian weights perform a distance-preserving embedding of the data, with a special treatment for in-class and out-of-class data. Similar points at the input of the network are likely to have a similar output. The theoretical analysis of deep networks here presented exploits tools used in the compressed sensing and dictionary learning literature, thereby making a formal connection between these important topics. The derived results allow drawing conclusions on the metric learning properties of the network and their relation to its structure, as well as providing bounds on the required size of the training set such that the training examples would represent faithfully the unseen data. The results are validated with state-of-the-art trained networks.

Subfields: Artificial Intelligence, Computational Mechanics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

Citations in the corpus

A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks (2016, background): “future avenues may utilize the intra-class variance: if the distance from an example to another of the same predicted class is abnormally high, it may be out-of-distribution (Giryes et al., 2015).”
Robust Large Margin Deep Neural Networks (2016, background): “The covering number of many structured low-dimensional data models can be bounded in terms of their “intrinsic” properties, for example: a Gaussian mixture model (GMM) with L Gaussians and covariance matrices of rank at most k leads to a covering number N (X ; d, ρ) = L(1 + 2/ρ) [41]; k-sparse signa”
Measuring and regularizing networks in function space (2018, background): “Giryes et al. (2016)).”
Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation (2019, background): “Other than the works discussed above, Li & Saad (2018); Giryes et al. (2016); Gabri et al. (2018); Reeves (2017); Fletcher & Rangan (2017) also considered neural networks with random weights.”
Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity (2016, background): “[19] demonstrated that for the MNIST and CIFAR-10 datasets the distances’ histogram of different examples barely changes when moving from the initial to the trained representation.”
How to Start Training: The Effect of Initialization and Architecture (2018, background): “For more on the Gaussian case, we also point the reader to [5].”
Building a Regular Decision Boundary with Deep Networks (2017, methodology): “Under this hypothesis, in a similar fashion as [10], we study the average intra-class distances.”
Mathematics of Deep Learning (2017, background): “, deep networks (almost) do not increase the intrinsic dimension of the data [56].”
Convolutional Neural Networks (2017, unknown): “On the other hand, Giryes et al. (2016) proved that CNNs with random Gaussian filters have metric preservation property, and they argued that the role of training is to select better hyperplanes discriminating classes by distorting boundary points among classes.”
Harnessing noisy Web images for deep representation (2017, background): “Giryes et al. (2015) proved that under random Gausian weights, deep neural networks are distance-preserving mapings with a special treatment for intra- and inter-class data.”
Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective (2021, background): “Figure 2: Example of linear regions divided by a ReLU network(1) The expressivity of a neural network indicates how complex the function it can represent (Hornik et al., 1989; Giryes et al., 2016).”
On the security relevance of weights in deep learning. (2020, background): “In accordance with this, weights are nowadays usually simply drawn from some zero-centered (and maybe cut-off) Gaussian distribution with appropriate variance [8], while the biases are often set to a constant.”
Efficient Representation of Low-Dimensional Manifolds using Deep Networks (2016, background): “(Giryes et al., 2016) shows that even a feed-forward neural network with random Gaussian weights embeds the input data in an output space while preserving distances between input items.”
Convolutional Neural Networks Analyzed via Convolutional Sparse Coding (2016, background): “Other works have studied the properties of deep and fully connected networks under the assumption of independent identically distributed random weights (Giryes et al., 2015; Saxe et al., 2013; Arora et al., 2014; Dauphin et al., 2014; Choromanska et al., 2015).”
DCFNet: Deep Neural Network with Decomposed Convolutional Filters (2018, background): “Deep networks with random weights have been studied in (Giryes et al., 2016), with proved representation stability.”