A topological insight into restricted Boltzmann machines

By Decebal Constantin Mocanu, Elena Mocanu, Phuong H. Nguyen, Madeleine Gibescu, Antonio Liotta. 2016

102 citations recorded by the source corpus.

Restricted Boltzmann Machines (RBMs) and models derived from them have been successfully used as basic building blocks in deep artificial neural networks for automatic features extraction, unsupervised weights initialization, but also as density estimators. Thus, their generative and discriminative capabilities, but also their computational time are instrumental to a wide range of applications. Our main contribution is to look at RBMs from a topological perspective, bringing insights from network science. Firstly, here we show that RBMs and Gaussian RBMs (GRBMs) are bipartite graphs which naturally have a small-world topology. Secondly, we demonstrate both on synthetic and real-world datasets that by constraining RBMs and GRBMs to a scale-free topology (while still considering local neighborhoods and data distribution), we reduce the number of weights that need to be computed by a few orders of magnitude, at virtually no loss in generative performance. Thirdly, we show that, for a fixed number of weights, our proposed sparse models (which by design have a higher number of hidden neurons) achieve better generative capabilities than standard fully connected RBMs and GRBMs (which by design have a smaller number of hidden neurons), at no additional computational costs.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

Online Contrastive Divergence with Generative Replay: Experience Replay without Storing Data (2016, background): “…models (Bengio, 2009), besides providing in an unsupervised manner good weights for the deep belief networks initialization (LeCun et al, 2015), have been shown to be very good density estimators and to have powerful generative capabilities (Salakhutdinov and Murray, 2008; Mocanu et al, 2016).”
SpaceNet: Make Free Space For Continual Learning (2020, background): “In [29], we have introduced the idea of training sparse neural networks from scratch for single task unsupervised learning.”
Sparse evolutionary deep learning with over one million artificial neurons on commodity hardware (2019, background): “Sparse evolutionary training Inspired by the fact that biological neural networks are prone to be sparse, rather than dense (Strogatz, 2001; Pessoa, 2014), there is an increasing interest in conceiving neural networks with a sparse topology (Mocanu et al., 2016; Yoon et al., 2017).”
Big IoT data mining for real-time energy disaggregation in buildings (2016, methodology): “FFW-CRBM is an extension of standard Restricted Boltzmann Machines (RBMs) [30], [31] (the basic block of deep learning models [32]) and consists of four layers of neurons, the history layer v<t which contains the last time series values, the present layer vt encoding the current time step (predictio”
Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training (2021, background): “, 2020)) or no over-parameterization (randomly-initialized static sparse training (Mocanu et al., 2016; Evci et al., 2019)) typically are not able to match the accuracy achieved by their dense counterpart.”