Learning Weight Uncertainty with Stochastic Gradient MCMC for Shape Classification

By Chunyuan Li, Andrew Stevens, Changyou Chen, Yunchen Pu, Zhe Gan, Lawrence Carin. 2016

55 citations recorded by the source corpus.

Learning the representation of shape cues in 2D & 3D objects for recognition is a fundamental task in computer vision. Deep neural networks (DNNs) have shown promising performance on this task. Due to the large variability of shapes, accurate recognition relies on good estimates of model uncertainty, ignored in traditional training of DNNs, typically learned via stochastic optimization. This paper leverages recent advances in stochastic gradient Markov Chain Monte Carlo (SG-MCMC) to learn weight uncertainty in DNNs. It yields principled Bayesian interpretations for the commonly used Dropout/DropConnect techniques and incorporates them into the SG-MCMC framework. Extensive experiments on 2D & 3D shape datasets and various DNN models demonstrate the superiority of the proposed approach over stochastic optimization. Our approach yields higher recognition accuracy when used in conjunction with Dropout and Batch-Normalization.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

Nonlinear Statistical Learning with Truncated Gaussian Graphical Models (2016, background): “The 20 NewsGroups data sets is composed of 18,845 documents, written with a vocabulary of 2,000 words, from 20 different groups, with the data partitioned into a training set of 11,315 documents and a testing set of 7,531 documents (Li et al., 2016).”
Uncertainty Estimations by Softplus normalization in Bayesian Convolutional Neural Networks with Variational Inference (2018, methodology): “Furthermore, structured variational approximations [23], auxiliary variables [24], and Stochastic Gradient MCMC [25] have been proposed to approximate the intractable posterior probability distribution.”
Towards Better Representations with Deep/Bayesian Learning (2018, methodology): “A popular method is to use Stochastic Gradient Markov Chain Monte Carlo (SG-MCMC) methods to generate posterior samples (Welling and Teh, 2011; Chen et al., 2014; Ding et al., 2014; Li et al., 2016a; Zhang et al., 2017c).”
Continuous-Time Flows for Efficient Inference and Density Estimation (2017, methodology): “In fact, CTF has been applied for scalable Bayesian sampling (Ding et al., 2014; Li et al., 2016a; Chen et al., 2016; Li et al., 2016b; Zhang et al., 2017).”
A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges (2020, unknown): “In addition, SGMCMC can be converged to the true posterior by decreasing the step sizes [70], [71].”
A survey of uncertainty in deep neural networks (2021, methodology): “Practically, SGMCMC techniques have been applied to shape classification and uncertainty quantification [188], empirically study and”
Scalable Bayesian Learning of Recurrent Neural Networks for Language Modeling (2016, methodology): “To help convergence, a momentum term has been introduced in SGHMC (Chen et al., 2014), a “thermostat” has been devised in SGNHT (Ding et al., 2014; Gan et al., 2015) and preconditioners have been employed in pS-GLD (Li et al., 2016a).”
Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning (2019, methodology): “We simply use the prescribed temperature of Li et al. (2016a) without tuning, but better results of the sampling methods can be achieved by tuning the temperature.”