Subdominant Dense Clusters Allow for Simple Learning and High Computational Performance in Neural Networks with Discrete Synapses.

By Carlo Baldassi, Alessandro Ingrosso, Carlo Lucibello, Luca Saglietti, Riccardo Zecchina. 2015

152 citations recorded by the source corpus.

We show that discrete synaptic weights can be efficiently used for learning in large scale neural systems, and lead to unanticipated computational performance. We focus on the representative case of learning random patterns with binary synapses in single layer networks. The standard statistical analysis shows that this problem is exponentially dominated by isolated solutions that are extremely hard to find algorithmically. Here, we introduce a novel method that allows us to find analytical evidence for the existence of subdominant and extremely dense regions of solutions. Numerical experiments confirm these findings. We also show that the dense regions are surprisingly accessible by simple learning protocols, and that these synaptic configurations are robust to perturbations and generalize better than typical solutions. These outcomes extend to synapses with multiple states and to deeper neural architectures. The large deviation measure also suggests how to design novel algorithmic schemes for optimization based on local entropy maximization.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Electrical and Electronic Engineering, Scaling & Learning Theory, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Electrical and Electronic Engineering

In Scaling & Learning Theory

In Statistical Learning Theory

Citations in the corpus

Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations (2016, background): “Other research (Baldassi et al., 2015) showed that full binary training and testing is possible in an array of committee machines with randomized input, where only one weight layer is being adjusted.”
XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks (2016, background): “[ 40] proposed fully binary training and testing in an array of committee machines with randomized input.”
Entropy-SGD: biasing gradient descent into wide valleys (2016, positive): “Motivated by the work of Baldassi et al. (2015) on shallow networks, instead of minimizing the original loss f (x), we propose to maximize F(x,γ) = log ∫ x′∈Rn exp ( − f (x′)− γ 2 ‖x− x′‖22 ) dx′.”
Binarized Neural Networks (2016, methodology): “Other research Baldassi et al. (2015) showed that full binary training and testing is possible in an array of committee machines with randomized input, where only one weight layer is being adjusted.”
Training Quantized Nets: A Deeper Understanding (2017, background): “The most extreme scenario of quantization is binarization, in which only 1-bit (two states) is used for weight representation [10, 5, 1, 3, 11, 12].”
Stochastic Gradient Descent Performs Variational Inference, Converges to Limit Cycles for Deep Networks (2017, background): “It was found that solutions of discrete learning problems that generalize well belong to dense clusters in the weight space (Baldassi et al., 2015; 2016).”
Non-vacuous Generalization Bounds at the ImageNet Scale: a PAC-Bayesian Compression Approach (2018, background): “This concept has also been widely studied (e.g., Langford, 2002; Langford & Caruana, 2002; Hinton & van Camp, 1993; Hochreiter & Schmidhuber, 1997; Baldassi et al., 2015; 2016; Chaudhari et al., 2017; Keskar et al., 2017; Dziugaite & Roy, 2017; Neyshabur et al., 2018).”
Unreasonable effectiveness of learning neural networks: From accessible states and robust ensembles to basic algorithmic schemes (2016, background): “Sets of conﬁgurations which are quite atypical for certain classes of algorithmic processes become typical for other processes.”
Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes (2017, background): “Similar works on this path can also be found in (Baldassi et al., 2015; 2016), where the discrete networks are considered.”
Entropy-SGD optimizes the prior of a PAC-Bayes bound: Data-dependent PAC-Bayes priors via differential privacy (2017, background): “…Entropy-SGLD only Step along stochastic local entropy ∇ 10: return w which several authors (Hinton and Camp, 1993; Hochreiter and Schmidhuber, 1997; Baldassi et al., 2015; Baldassi et al., 2016) where w 1 , w 2 ,... are (approximately) i.i.d. samples from G w , S γ , τ and α ∈ ( 0 , 1 ) deﬁnes a…”
Shaping the learning landscape in neural networks around wide flat minima (2019, methodology): “While this phenomenon was recently disclosed for the case of discrete weights (3, 4), here we demonstrate that it is present also in nonconvex ANN with continuous weights.”
Entropy-SGD optimizes the prior of a PAC-Bayes bound: Generalization properties of Entropy-SGD and data-dependent priors (2017, background): “…argue informally that maximizing the local entropy leads to “ﬂat minima” in the empirical risk surface, which several authors (Hinton & van Camp, 1993; Hochre-iter & Schmidhuber, 1997; Baldassi et al., 2015; 2016) w (cid:48) , µ ← w 3: for i ∈ { 1 ,..., L } do (cid:46) Run SGLD for L iterations.”
Local entropy as a measure for sampling solutions in constraint satisfaction problems (2015, methodology): “…has shown that indeed such sub-dominant ultra-dense regions exist at least up to a critical value of the parameter α = , that these kind of solutions exhibit diﬀerent properties with respect to typical equilibrium solutions, and that heuristic solvers typically ﬁnd a solution in such regions [7].”
Machine learning and the physical sciences (2019, background): “Another intriguing line of work that relates learning in neural networks to properties of landscapes is explored in (Baldassi et al., 2016, 2015).”