A Deep and Autoregressive Approach for Topic Modeling of Multimodal Data

By Yin Zheng, Yujin Zhang, Hugo Larochelle. 2014

63 citations recorded by the source corpus.

Topic modeling based on latent Dirichlet allocation (LDA) has been a framework of choice to deal with multimodal data, such as in image annotation tasks. Another popular approach to model the multimodal data is through deep neural networks, such as the deep Boltzmann machine (DBM). Recently, a new type of topic model called the Document Neural Autoregressive Distribution Estimator (DocNADE) was proposed and demonstrated state-of-the-art performance for text document modeling. In this work, we show how to successfully apply and extend this model to multimodal data, such as simultaneous image classification and annotation. First, we propose SupDocNADE, a supervised extension of DocNADE, that increases the discriminative power of the learned hidden topic features and show how to employ it to learn a joint representation from image visual words, annotation words and class label information. We test our model on the LabelMe and UIUC-Sports data sets and show that it compares favorably to other topic models. Second, we propose a deep extension of our model and provide an efficient way of training the deep model. Experimental results show that our deep model outperforms its shallow version and reaches state-of-the-art performance on the Multimedia Information Retrieval (MIR) Flickr data set.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Mingkui Tan
2. Jian Chen
3. Yong Guo
4. Peilin Zhao
5. Hanning Zhou
6. Bangsheng Tang
7. Qi Chen
8. Junzhou Huang

In Computer Vision and Pattern Recognition

1. Ming-Hsuan Yang
2. Jian Chen
3. Mingkui Tan
4. Yong Guo
5. Jin-shan Pan
6. Luc Van Gool
7. Junzhou Huang
8. Deqing Sun

Citations in the corpus

• A Neural Autoregressive Approach to Collaborative Filtering (2016, background): “(2014) proposed an efficient deep extension to original NADE (Larochelle & Murray, 2011) for binary vector observations, which inspires other related deep model (Zheng et al., 2015).”
• Learning to Super-Resolve Blurry Face and Text Images (2017, background): “Deep learning achieves promising performance on many applications [21, 52, 1, 14].”
• NAT: Neural Architecture Transformer for Accurate and Compact Architectures (2019, background): “…neural networks (DNNs) [25] have been producing state-of-the-art results in many challenging tasks including image classification [12, 23, 43, 60, 61, 18, 11, 56], face recognition [39, 44, 59], brain signal processing [34, 35], video analysis [48, 53, 52] and many other areas [58, 57, 24, 3,…”
• Breaking the Curse of Space Explosion: Towards Efficient NAS with Curriculum Search (2020, background): “…many challenging tasks including image classification (Krizhevsky et al., 2012; Liu et al., 2020; Zheng et al., 2015; Guo et al., 2020a), semantic segmentation (Shelhamer et al., 2017; Chen et al., 2018; Huang et al., 2019), and many other areas (Zheng et al., 2016b;a; Jiang *Equal contribution .”
• Variational Deep Embedding: An Unsupervised and Generative Approach to Clustering (2016, unknown): “tensively used, learning good representations most suitable for clustering tasks is left largely unexplored. Recently, deep learning has achieved widespread success in numerous machine learning tasks [9,11,3,36,35,37,1, 7], where learning good representations by deep neural networks (DNN) lies in th”