Bayesian dark knowledge

By Masashi Sugiyama, Roman Garnett. 2015

271 citations recorded by the source corpus.

We consider the problem of Bayesian parameter estimation for deep neural networks, which is important in problem settings where we may have little data, and/ or where we need accurate posterior predictive densities p(y|x, D), e.g., for applications involving bandits or active learning. One simple approach to this is to use online Monte Carlo methods, such as SGLD (stochastic gradient Langevin dynamics). Unfortunately, such a method needs to store many copies of the parameters (which wastes memory), and needs to make predictions using many versions of the model (which wastes time). We describe a method for "distilling" a Monte Carlo approximation to the posterior predictive density into a more compact form, namely a single deep neural network. We compare to two very recent approaches to Bayesian neural networks, namely an approach based on expectation propagation [HLA15] and an approach based on variational Bayes [BCKW15]. Our method performs better than both of these, is much simpler to implement, and uses less computation at test time.

Subfields: Artificial Intelligence, Statistics and Probability, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

In Probabilistic & Bayesian Methods

Citations in the corpus

Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding (2019, background): “…a process of distilling or transferring the knowledge from a (set of) large, cumbersome model(s) to a lighter, easier-to-deploy single model, without signiﬁcant loss in performance (Bucilu et al., 2006; Hinton et al., 2015; Balan et al., 2015; Ba et al., 2016; Chen et al., 2015; Tan et al., 2019).”
Self-Training With Noisy Student Improves ImageNet Classification (2019, methodology): “As we use soft targets, our work is also related to methods in Knowledge Distillation [7, 3, 26, 16].”
Large Language Models Can Self-Improve (2022, neutral): “A key detail is to learn from soft targets instead of hard predicted labels, as softmax outputs with a high temperature reveal more detailed relative class likelihoods, col-loquially known as dark knowledge (Hinton et al., 2015; Korattikara Balan et al., 2015).”
A Theoretically Grounded Application of Dropout in Recurrent Neural Networks (2015, background): “Recent results at the intersection of Bayesian research and deep learning offer interpretation of common deep learning techniques through Bayesian eyes [11–16].”
Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles (2016, methodology): “…(Blundell et al., 2015; Graves, 2011; Louizos and Welling, 2016), assumed density ﬁltering (Hern´andez-Lobato and Adams, 2015), expectation propagation (Hasenclever et al., 2015; Li et al., 2015) or stochastic gradient Langevin diffusion methods (Korattikara et al., 2015; Welling and Teh, 2011).”
Beyond Sparsity: Tree Regularization of Deep Models for Interpretability (2017, methodology): “interpretability, there are some related threads. The ﬁrst is model compression, which trains smaller models that perform similarly to large, blackbox models (e.g. (?; Hinton, Vinyals, and Dean 2015; Balan et al. 2015; Han et al. 2015)). Other efforts speciﬁ- cally train very sparse networks via L1 ”
Distilling Knowledge from Deep Networks with Applications to Healthcare Domain (2015, background): “[26] takes a Bayesian approach for distilling knowledge from a teacher neural network to train a student neural network.”
Bayesian Model-Agnostic Meta-Learning (2018, background): “It brings us not only robustness to overﬁtting but also numerous beneﬁts such as improved prediction accuracy by Bayesian ensembling (Balan et al., 2015), active learning (Gal et al., 2016), and principled/safe exploration in reinforcement learning (Houthooft et al., 2016).”
FOSTER: Feature Boosting and Compression for Class-Incremental Learning (2022, background): “Knowledge distillation [19] aims to transfer dark knowledge [24] from the teacher to the student by encouraging the outputs of the student model to approximate the outputs of the teacher model [27].”
Multiplicative Normalizing Flows for Variational Bayesian Neural Networks (2017, methodology): “…using either Markov Chain Monte Carlo (MCMC) with Hamiltonian Dynamics (Neal, 1995), distilling SGD with Langevin Dynamics (Welling & Teh, 2011; Korattikara et al., 2015) or deterministic techniques such as the Laplace Approximation (MacKay, 1992), Expectation Propagation (Hern ´ andez-Lobato…”
Lightweight Probabilistic Deep Networks (2018, background): “[33] train a student to reproduce the behavior of a teacher network trained in the full Bayesian setting.”
Natural-Parameter Networks: A Class of Probabilistic Neural Networks (2016, methodology): “d tasks. Thanks to the properties of the exponential family [3, 19], distributions in NPN are deﬁned by the corresponding natural parameters which can be learned efﬁciently by backpropagation. Unlike [4, 1], NPN explicitly propagates the estimates of uncertainty back and forth in deep networks. This”
Assumed Density Filtering Methods for Learning Bayesian Neural Networks (2016, methodology): “Promising recent work has focused on improving both scalability and accuracy (Soudry, Hubara, and Meir 2014; Hernández-Lobato and Adams 2015; Balan et al. 2015; Blundell et al. 2015).”
Task Agnostic Continual Learning Using Online Variational Bayes (2018, methodology): “However, in contrast to the SGLD approach, we are not required to store all copies of the networks for inference ([3] had to use distillation to overcome this), but only two parameters for each weight (μ and σ).”
Learning and Policy Search in Stochastic Dynamical Systems with Bayesian Neural Networks (2016, background): “Recently, numerous works have developed efficient inference approaches for BNNs, including variational methods [2, 6], probabilistic backpropagation [9] and Markov Chain Monte Carlo methods [1, 25].”
Task-Agnostic Continual Learning Using Online Variational Bayes With Fixed-Point Updates (2020, background): “…Adams, 2015), approximate power expectation propagation (Hernández-Lobato et al., 2016), stochastic Langevin gradient descent (Welling & Teh, 2011; Balan, Rathod, Murphy, & Welling, 2015), incremental moment matching (Lee, Kim, Ha, & Zhang, 2017), or variational Bayes (Graves, 2011; Blundell et…”