Learning with Hierarchical-Deep Models

By Ruslan Salakhutdinov, Josh Tenenbaum, Antonio Torralba. 2013

247 citations recorded by the source corpus.

We introduce HD (or “Hierarchical-Deep”) models, a new compositional learning architecture that integrates deep learning models with structured hierarchical Bayesian (HB) models. Specifically, we show how we can learn a hierarchical Dirichlet process (HDP) prior over the activities of the top-level features in a deep Boltzmann machine (DBM). This compound HDP-DBM model learns to learn novel concepts from very few training example by learning low-level generic features, high-level features that capture correlations among low-level features, and a category hierarchy for sharing priors over the high-level features that are typical of different kinds of concepts. We present efficient learning and inference algorithms for the HDP-DBM model and show that it is able to learn new concepts from very few examples on CIFAR-100 object recognition, handwritten character recognition, and human motion capture datasets.

Subfields: Computer Vision and Pattern Recognition, Model Architectures, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Model Architectures

In Signal Processing

Citations in the corpus

Conditional Image Generation with PixelCNN Decoders (2016, background): “In the future it might be interesting to try and generate new images with a certain animal or object solely from a single example image [21, 22].”
Space-Time Representation of People Based on 3D Skeletal Data: A Review (2016, background): “[163] proposed a compositional learning architecture that integrates deep learning models with structured hierarchical Bayesian models.”
RGB-D-based Human Motion Recognition with Deep Learning: A Survey (2017, background): “Salakhutdinov et al. (2013) proposed a new compositional learning architecture that integrates deep learning models with structured hierarchical Bayesian models.”
Fast Adaptation in Generative Models with Generative Matching Networks (2016, methodology): “Salakhutdinov et al. (2013) combined deep Boltzmann machine (DBM) with nested Dirichlet process (nDP) in a Hierarchical-Deep (HD) model.”
Towards more human-like concept learning in machines : compositionality, causality, and learning-to-learn (2014, methodology): “Causality was not tested, and future work should compare against deep learning approaches such as the HD model (Salakhutdinov et al., 2013).”
One-Shot Learning in Discriminative Neural Networks (2017, result): “Similar to [Salakhutdinov et al., 2013], we could place a Dirichlet Process prior on the weight space to automatically infer the number of supercategory clusters.”
Reconciling meta-learning and continual learning with online mixtures of tasks (2018, background): “Closest to our proposed approach is early work on hierarchical Bayesian multi-task learning with neural networks that places a prior only on the output layer [24, 2, 46, 50].”
A Simple Exponential Family Framework for Zero-Shot Learning (2017, methodology): “In few-shot learning, we assume that a very small number of labeled examples may also be available for the unseen classes [23,17].”
Zero-Shot Learning via Class-Conditioned Deep Generative Models (2017, methodology): “A As can be seen, even with as few as 2 or 5 additional labeled examples per class, the FSL signiﬁcantly improves over ZSL.”
Teaching Compositionality to CNNs (2017, background): “representations from the ground up [34, 45, 43, 47], our approach does not mandate any particular network architecture or parameterization – instead, it comes in the form of a modified training objective that can be applied to teach any standard CNN about compositionality in a soft manner.”
Human-level concept learning through probabilistic program induction (2015, background): “There was an average ID level difference of 9% across the two phases, which approached significance (t(29) = 2.”