An exact mapping between the Variational Renormalization Group and Deep Learning

By Pankaj Mehta, David J. Schwab. 2014

344 citations recorded by the source corpus.

Deep learning is a broad set of techniques that uses multiple layers of representation to automatically learn relevant features directly from structured data. Recently, such techniques have yielded record-breaking results on a diverse set of difficult machine learning tasks in computer vision, speech recognition, and natural language processing. Despite the enormous success of deep learning, relatively little is understood theoretically about why these techniques are so successful at feature learning and compression. Here, we show that deep learning is intimately related to one of the most important and successful techniques in theoretical physics, the renormalization group (RG). RG is an iterative coarse-graining scheme that allows for the extraction of relevant features (i.e. operators) as a physical system is examined at different length scales. We construct an exact mapping from the variational renormalization group, first introduced by Kadanoff, and deep learning architectures based on Restricted Boltzmann Machines (RBMs). We illustrate these ideas using the nearest-neighbor Ising Model in one and two-dimensions. Our results suggests that deep learning algorithms may be employing a generalized RG-like scheme to learn relevant features from data.

Subfields: Atomic and Molecular Physics, and Optics, Materials Chemistry, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Atomic and Molecular Physics, and Optics

In Materials Chemistry

In Statistical and Nonlinear Physics

Citations in the corpus

Geometric Deep Learning: Going beyond Euclidean data (2016, background): “These statistical properties have been related to physics [16] and formalized in specific classes of convolutional neural networks (CNNs) [17], [18], [19].”
Deep learning and the information bottleneck principle (2015, background): “Another outcome of this representation is a possible explanation of the layered architecture of the network, different from the one suggested in [4].”
Quantum machine learning (2016, positive): “Mehta and Schwab [93] merges ideas of mapping from the variational renormalization group, ﬁrst introduced by Kadanoﬀ [95], and deep learning models based on restricted Boltzmann machines so that the results indicate that deep learning algorithms might be viewed as employing a generalized…”
Colloquium: Criticality and dynamical scaling in living systems (2017, background): “…deep learning machines (LeCun et al., 2015) may rely on some form of intrinsic scale invariance or even criticality (Lin and Tegmark, 2017; Mehta and Schwab, 2014; Oprisa and Toth, 2017a,b; Ringel and de Bem, 2018; Song et al., 2017), opening exciting research avenues to understand how…”
A Mathematical Motivation for Complex-Valued Convolutional Networks (2015, background): “A related, more sophisticated connection (to renormalization group theory) is given by Mehta and Schwab (2014).”
Doing the Impossible: Why Neural Networks Can Be Trained at All (2018, background): “It is also related to the recently pointed connection between the Variational Renormalization Group and Restricted Boltzmann Machines (Mehta and Schwab, 2014) as well as the Information Bottleneck analysis of deep neural networks (Schwartz-Ziv and Tishby, 2017).”
A high-bias, low-variance introduction to Machine Learning for physicists (2018, positive): “…features is reminiscent of ideas like the renormalization group (RG) in physics where the RG flows separate out relevant and irrelevant directions, and certain unsupervised deep learning architectures have a natural interpretation in terms of variational RG schemes (Mehta and Schwab, 2014).”
Flow of Information in Feed-Forward Deep Neural Networks (2016, methodology): “Mehta and Schwab shown in [10] that deep learning is related to renormalization groups in theoretical physics and provide a mapping between deep learning methods and variational renormalization groups using Restricted Boltzmann Machines.”
Machine learning quantum phases of matter beyond the fermion sign problem (2016, background): “On a conceptual level, parallels between deep learning and renormalization group techniques have been explored [19, 20], while on a more practical level machine learning algorithms have been applied to model potential energy surfaces [21], relaxation in glassy liquids [22] or the identiﬁcation of…”
On Interpretability of Artificial Neural Networks: A Survey (2020, background): “However, in their study the network structure was simplified in concatenating two ReLU units into a linear unit such that the nonlinearity from ReLU units could be circumvented.”
Unsupervised learning of phase transitions: from principal component analysis to variational autoencoders (2017, background): “Furthermore, since there exists a connection between deep neural networks and renormalization group [38], it may be helpful to employ deep convolutional autoencoders to further expose this connection.”
Machine Learning Phases of Strongly Correlated Fermions (2016, neutral): “Early applications of machine learning ideas in condensed matter physics focused on their connection to renormalization group methods [2], obtaining the Green ’ s function of the Anderson impurity model [3], categorizing real materials [4 – 7], or learning ground states and thermo-dynamics of…”
Neural Network Renormalization Group (2018, methodology): “The NeuralRG approach is closer in spirit to the original proposal based on Bayesian net [20] than more recent discussions on Boltzmann Machines [21, 23] and Principal Component Analysis [22].”
Flow of Renyi information in deep neural networks (2016, background): “[1] showed that there is an exact mapping between the variational renormalization group (RG) and DNNs based on restricted Boltzmann machines (RBM).”
Quantum convolutional neural networks (2018, methodology): “In case (3), the optimized QEC code performs substantially better than known codes (Fig.”
Quantum Entanglement in Neural Network States (2017, background): “phases [32]), modeling thermodynamic observables [36], constructing decoders for topological codes [39], accelerating Monte Carlo simulations [41, 42], and establishing connections to renormalization group techniques [45, 46], etc.”
Parameterized quantum circuits as machine learning models (2019, background): “At a theoretical level, the hierarchical representation used in deep learning for feature extraction can be mapped to the renormalization group [45], and entanglement has shown to be a relevant quantity to measure how deep networks express correlations in data [46].”