Efficient and optimal binary Hopfield associative memory storage using minimum probability flow

By Christopher J. Hillar, Jascha Narain Sohl-Dickstein, Kilian Koepsell. 2012

25 citations recorded by the source corpus.

We present an algorithm to store binary memories in a Hopfield neural network using minimum probability flow, a recent technique to fit parameters in energy- based probabilistic models. In the case of memories without noise, our algorithm provably achieves optimal pattern storage (which we show is at least one pattern per neuron) and outperforms classical methods both in speed and memory recov- ery. Moreover, when trained on noisy or corrupted versions of a fixed set of binary patterns, our algorithm finds networks which correctly store the originals. We also demonstrate this finding visually with the unsupervised storage and clean-up of large binary fingerprint images from significantly corrupted samples. 1 Introduction. In 1982, motivated by neural modeling work of (1) and the Ising spin glass model from statistical physics (2), Hopfield introduced a method for the storage and retrieval of binary patterns in an auto-associative neural-network (3). Even today, this model and its various exten- sions (4, 5) provide a plausible mechanism for memory formation in the brain. However, existing techniques for training Hopfield networks suffer either from limited pattern capacity or excessive training time, and they exhibit poor performance when trained on unlabeled, corrupted memories. Our main theoretical contributions here are the introduction of a tractable and neurally-plausible algorithm for the optimal storage of patterns in a Hopfield network, a proof that the capacity of such a network is at least one pattern per neuron, and a novel local learning rule for training neural networks. Our approach is inspired by minimum probability flow (6), a recent technique for fit- ting probabilistic models that avoids computations with a partition function, the usually intractable normalization constant of a parameterized probability distribution. We also present several experimental results. When compared with standard techniques for Hop- field pattern storage, our method is shown to be superior in efficiency and generalization. Another finding is that our algorithm can store many patterns in a Hopfield network from highly corrupted (unlabeled) samples of them. This discovery is also corroborated visually by the storage of 64 64 binary images of human fingerprints from highly corrupted versions, as in Fig. 1.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Probabilistic & Bayesian Methods

Citations in the corpus

Fast large-scale optimization by unifying stochastic gradient and quasi-Newton methods (2013, methodology): “• An Ising model / Hopﬁeld network trained using code from (Hillar et al., 2012) implementing MPF (Sohl-Dickstein et al., 2011b;a).”
Neural Empirical Bayes (2019, background): “These limitations are discussed in depth in the literature, most recently in (Hillar et al., 2012; Krotov and Hopfield, 2016; Chaudhuri and Fiete, 2017), where the authors propose alternative solutions we highlight here: (Hillar et al.”