Constructing Hierarchical Image-tags Bimodal Representations for Word Tags Alternative Choice

By Fangxiang Feng, Ruifan Li, Xiaojie Wang. 2013

12 citations recorded by the source corpus.

This paper describes our solution to the multi-modal learning challenge of ICML. This solution comprises constructing three-level representations in three consecutive stages and choosing correct tag words with a data-specific strategy. Firstly, we use typical methods to obtain level-1 representations. Each image is represented using MPEG-7 and gist descriptors with additional features released by the contest organizers. And the corresponding word tags are represented by bag-of-words model with a dictionary of 4000 words. Secondly, we learn the level-2 representations using two stacked RBMs for each modality. Thirdly, we propose a bimodal auto-encoder to learn the similarities/dissimilarities between the pairwise image-tags as level-3 representations. Finally, during the test phase, based on one observation of the dataset, we come up with a data-specific strategy to choose the correct tag words leading to a leap of an improved overall performance. Our final average accuracy on the private test set is 100%, which ranks the first place in this challenge.

Subfields: Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Challenges in representation learning: A report on three machine learning contests (2013, background): “al matching. The optimal matching constructed in this way obtained 100% accuracy. The contest ended in a three-way tie with 100% test accuracy. The winners were \RBM" (Yichuan Tang), \MMDL" [19] (Fangxiang Feng, Ruifan Li, and Xiaojie Wang), and \AlbinoSnowman" (John Park). RBM won th”
Learning Grounded Meaning Representations with Autoencoders (2014, background): “…focused on projecting words and images into a common space using a variety of deep learning methods ranging from deep and restricted Boltzman machines (Srivastava and Salakhutdinov, 2012; Feng et al., 2013), to au-toencoders (Wu et al., 2013), and recursive neural networks (Socher et al., 2013b).”
Learning Image Embeddings using Convolutional Neural Networks for Improved Multi-Modal Semantics (2014, background): “Other examples that apply multi-modal deep learning use restricted Boltzmann machines (Sri-vastava and Salakhutdinov, 2012; Feng et al., 2013), auto-encoders (Wu et al., 2013) or recursive neural networks (Socher et al., 2014).”
Gated Multimodal Units for Information Fusion (2017, methodology): “Other similar strategies propose to fusion modalities using neural network architectures (Andrew et al., 2013; Feng et al., 2013; Kang et al., 2012; Kiros et al., 2014a; Lu et al., 2014; Mao et al., 2014; Tu et al., 2014; Wu et al., 2013) with two input layers separately and including a ﬁnal…”