Multimodal Deep Learning

By Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, Andrew Y. Ng. 2011

3,550 citations recorded by the source corpus.

Deep networks have been successfully applied to unsupervised feature learning for single modalities (e.g., text, images or audio). In this work, we propose a novel application of deep networks to learn features over multiple modalities. We present a series of tasks for multimodal learning and show how to train deep networks that learn features to address these tasks. In particular, we demonstrate cross modality feature learning, where better features for one modality (e.g., video) can be learned if multiple modalities (e.g., audio and video) are present at feature learning time. Furthermore, we show how to learn a shared representation between modalities and evaluate it on a unique task, where the classifier is trained with audio-only data but tested with video-only data and vice-versa. Our models are validated on the CUAVE and AVLetters datasets on audio-visual speech classification, demonstrating best published visual speech classification on AVLetters and effective shared representation learning.

Subfields: Artificial Intelligence, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Yoshua Bengio
  2. Mohit Bansal
  3. Jianfeng Gao
  4. Quoc V. Le
  5. Sergey Levine
  6. Percy Liang
  7. Trevor Darrell
  8. Dacheng Tao

In Multimodal Learning

  1. Trevor Darrell
  2. Marcus Rohrbach
  3. Devi Parikh
  4. Mohit Bansal
  5. Andrew Zisserman
  6. Qi Wu
  7. Zhe Gan
  8. Kristen Grauman

Citations in the corpus