A Learning Algorithm for Boltzmann Machines

By David H. Ackley, Geoffrey E. Hinton, Terrence J. Sejnowski. 1985

Subfields: Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Probabilistic & Bayesian Methods

Citations in the corpus

The Curious Case of Neural Text Degeneration (2019, background): “Another common approach to sampling-based generation is to shape a probability distribution through temperature (Ackley et al., 1985).”
Self-Consistency Improves Chain of Thought Reasoning in Language Models (2022, methodology): “Multiple decoding strategies for language models have been proposed in the literature, e.g., temperature sampling (Ackley et al., 1985; Ficler & Goldberg, 2017), top-k sampling (Fan et al., 2018; Holtzman et al., 2018; Radford et al., 2019), nucleus sampling (Holtzman et al., 2020), minimum Bayes…”
Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation (2020, background): “It has been heavily exploited in modern neural networks [50,49] due to its efficiency and generalization ability, in comparison to fully connected models [2].”
Diffusion Models Beat GANs on Image Synthesis (2021, background): “Most likelihood-based models also allow for low-temperature sampling [1], which provides a natural way to emphasize modes of the data distribution (see Appendix I).”
High Fidelity Visualization of What Your Self-Supervised Representation Knows About (2021, background): “These include probabilistic latent variable models and variants of auto-encoders (Ackley et al., 1985; Hinton et al., 2006; Salakhutdinov et al., 2007; Vincent et al., 2008; Kingma & Welling, 2014; Rezende et al., 2014), that are traditionally put under the broad umbrella term of unsupervised learni”
Rationale-Augmented Ensembles in Language Models (2022, methodology): “For sampling in the language model, we use temperature sampling (Ackley et al., 1985; Ficler & Goldberg, 2017) with temperature T = 0 .”
Generative Moment Matching Networks (2015, methodology): “Thus, with GMMNs it is easy to quickly draw independent random samples, as opposed to expensive MCMC procedures that are necessary in other models such as Boltzmann machines (Ackley et al., 1985; Hinton, 2002; Salakhutdinov & Hinton, 2009).”
NIPS 2016 Tutorial: Generative Adversarial Networks (2016, background): “Boltzmann machines (Fahlman et al., 1983; Ackley et al., 1985; Hinton et al., 1984; Hinton and Sejnowski, 1986) are a family of generative models that rely on Markov chains both to train the model or to generate a sample from the model.”
On the Binding Problem in Artificial Neural Networks (2020, background): “Early work includes Hopfield networks (Hopfield, 1982), Boltzmann machines (Ackley et al., 1985), and associative memory (Kohonen, 1989).”
Online Contrastive Divergence with Generative Replay: Experience Replay without Storing Data (2016, background): “Narasimhan et al (2015) proposed a form of re-sampling in the context of DRL, separating the learner experience into two parts: one for the positive rewards and one for the negative.”
Lightweight Probabilistic Deep Networks (2018, background): “Perhaps among the ﬁrst probabilistic neural networks are the works relating undirected graphical models to neural networks via Boltz-mann machines [3, 24]; similarly, links are established for deep belief nets (DBN) via sigmoid belief nets [23, 41].”
Deep Directed Generative Models with Energy-Based Probability Estimation (2016, methodology): “The energy function is typically used in order to parametrize undirected graphical models, such as Boltzmann machines [1], in the form of a Boltzmann distribution with an appropriate normalization factor.”
Minimum Probability Flow Learning (2009, methodology): “We demonstrate that learning under this framework is effective and fast in a number of cases: Ising models (Brush, 1967; Ackley et al., 1985), deep belief networks (Hinton et al.”
Wasserstein Introspective Neural Networks (2017, background): “Before the deep learning era [17], generative modeling had been an area with a steady pace of development [1, 5, 54, 36, 49, 53].”
Energy-based Out-of-distribution Detection (2020, background): “Energy-based learning Energy-based machine learning models date back to Boltzmann machines [1, 35], networks of units with an energy defined for the overall network.”
Implicit Generation and Generalization in Energy-Based Models (2019, methodology): “Ackley et al. [1985], Hinton [2006], Salakhutdinov and Hinton [2009] proposed latent based EBMs where energy is represented as a composition of latent and observable variables.”
Synthesizing Programs for Images using Reinforced Adversarial Learning (2018, background): “…and learning feed-forward or recurrent procedures for image generation (LeCun et al., 2015; Hinton & Salakhutdinov, 2006; Goodfellow et al., 2014; Ackley et al., 1987; Kingma & Welling, 2013; Oord et al., 2016; Kulkarni et al., 2015b; Eslami et al., 2016; Reed et al., 2017; Gregor et al., 2015).”
Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks (2018, background): “…& Szegedy, 2015a), orthogonal matrix initialization (Saxe et al., 2013), random walk initialization (Sussillo & Abbott, 2014), composition kernels (Daniely et al., 2016), or residual network architectures (He et al., 2015) all share a common goal of stabilizing gradients and improving training…”
Toward an Integration of Deep Learning and Neuroscience (2016, background): “…inference in certainclassesofsymmetricnetwork(Hopﬁeld,1982),researchershavediscovered networkswithinherentdynamicsthatimplicitlyoptimizescertainobjectivesevenwhiletheconnectionweightsareﬁxed,suchasstatisticalreconstruction oftheinputviastochasticrelaxationinBoltzmannmachines(Ackleyetal., 1958 ).”