Associative Multichannel Autoencoder for Multimodal Word Representation

By Shaonan Wang, Jiajun Zhang, Chengqing Zong. 2018

26 citations recorded by the source corpus.

In this paper we address the problem of learning multimodal word representations by integrating textual, visual and auditory inputs. Inspired by the re-constructive and associative nature of human memory, we propose a novel associative multichannel autoencoder (AMA). Our model first learns the associations between textual and perceptual modalities, so as to predict the missing perceptual information of concepts. Then the textual and predicted perceptual representations are fused through reconstructing their original and associated embeddings. Using a gating mechanism our model assigns different weights to each modality according to the different concepts. Results on six benchmark concept similarity tests show that the proposed method significantly outperforms strong unimodal baselines and state-of-the-art multimodal models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jiajun Zhang
2. Chengqing Zong
3. Jing Gao
4. Peng Li
5. Zhikui Chen
6. Jianing Zhang
7. Jingyuan Sun
8. Shaonan Wang

In Computer Vision and Pattern Recognition

1. Jing Gao
2. Peng Li
3. Zhikui Chen
4. Jianing Zhang
5. Chengqing Zong
6. Jingyuan Sun
7. Shaonan Wang
8. Jiajun Zhang

Citations in the corpus

• Distill and Replay for Continual Language Learning (2020, background): “However, the currently dominant paradigm of machine language learning is still training a model on a static dataset to achieve satisfactory performance on that particular task (Wang et al., 2018; Ostapenko et al., 2019).”
• A Survey on Deep Learning for Multimodal Data Fusion (2020, background): “For example, Wang, Zhang, and Zong (2018) designed a multimodal stacked autoencoder for feature learning of words, which the association and gating mechanisms are adopted to improve the word features.”