Natural-Parameter Networks: A Class of Probabilistic Neural Networks

By Hao Wang, Xingjian Shi, Dit-Yan Yeung. 2016

89 citations recorded by the source corpus.

Neural networks (NN) have achieved state-of-the-art performance in various applications. Unfortunately in applications where training data is insufficient, they are often prone to overfitting. One effective way to alleviate this problem is to exploit the Bayesian approach by using Bayesian neural networks (BNN). Another shortcoming of NN is the lack of flexibility to customize different distributions for the weights and neurons according to the data, as is often done in probabilistic graphical models. To address these problems, we propose a class of probabilistic neural networks, dubbed natural-parameter networks (NPN), as a novel and lightweight Bayesian treatment of NN. NPN allows the usage of arbitrary exponential-family distributions to model the weights and neurons. Different from traditional NN and BNN, NPN takes distributions as input and goes through layers of transformation before producing distributions to match the target output distributions. As a Bayesian treatment, efficient backpropagation (BP) is performed to learn the natural parameters for the distributions over both the weights and neurons. The output distributions of each layer, as byproducts, may be used as second-order representations for the associated tasks such as link prediction. Experiments on real-world datasets show that NPN can achieve state-of-the-art performance.

Subfields: Artificial Intelligence, Model Architectures, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Probabilistic & Bayesian Methods

Citations in the corpus

A General Framework for Uncertainty Estimation in Deep Learning (2019, background): “[9] propose natural parameter networks, which model inputs, parameters, nodes, and targets by Gaussian distributions.”
Lightweight Probabilistic Deep Networks (2018, background): “[51] propose natural parameter networks (NPN), which treat the inputs, targets, weights, and neurons probabilistically by exponential-family distributions, e.g . assuming a Gaussian distribution for both weights and activations.”
Ieee Transactions on Knowledge and Data Engineering 1 towards Bayesian Deep Learning: a Framework and Some Existing Methods (unknown): “Fortunately, some recent advances in this direction [2], [9], [23], [34], [66] seem to shed light on the practical adoption of Bayesian neural networks 1 .”
A Survey on Bayesian Deep Learning (2016, methodology): “Fortunately, some recent advances in this direction [2, 9, 31, 39, 58, 119, 121] seem to shed light2 on the practical adoption of Bayesian neural network3.”
Continuously Indexed Domain Adaptation (2020, methodology): “For example, one can use a probabilistic encoder parameterized by a natural-parameter network (Wang et al., 2016) and generate z using the reparameterization trick (Kingma & Welling, 2013).”
ProbGAN: Towards Probabilistic GAN with Theoretical Guarantees (2018, methodology): “In the spirit of boosting algorithm, Wang et al. (2016b) propose to progressively train new generator using a subset of training data that are not well captured by previous generators, while Tolstikhin et al. (2017) further propose a more robust mechanism to reweight samples in the training set for…”
Relational Deep Learning: A Deep Latent Variable Model for Link Prediction (2017, methodology): “With the rapid growth of social network services (SNS) and other Internet applications, network data have become very pervasive (Wang et al. 2016).”
Towards Bayesian Deep Learning: A Survey (2016, methodology): “with reasonable time complexity. This line of work is pioneered by [42, 72, 80], but it has not been widely adopted due to its lack of scalability. Fortunately, some recent advances in this direction [2, 9, 31, 39, 58, 119, 121] seem to shed light2 on the practical adoption of Bayesian neural networ”
Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Vision (2019, background): “To tackle this problem, a wide variety of approximations have been explored [38, 22, 4, 48, 7, 21, 47], but only a small number have been demonstrated to be applicable even to the large-scale DNN models commonly employed in real-world computer vision tasks.”