Multimodal Deep Learning
By Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, Andrew Y. Ng. 2011
3,550 citations recorded by the source corpus.
Deep networks have been successfully applied to unsupervised feature learning for single modalities (e.g., text, images or audio). In this work, we propose a novel application of deep networks to learn features over multiple modalities. We present a series of tasks for multimodal learning and show how to train deep networks that learn features to address these tasks. In particular, we demonstrate cross modality feature learning, where better features for one modality (e.g., video) can be learned if multiple modalities (e.g., audio and video) are present at feature learning time. Furthermore, we show how to learn a shared representation between modalities and evaluate it on a unique task, where the classifier is trained with audio-only data but tested with video-only data and vice-versa. Our models are validated on the CUAVE and AVLetters datasets on audio-visual speech classification, demonstrating best published visual speech classification on AVLetters and effective shared representation learning.
Subfields: Artificial Intelligence, Multimodal Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Mohit Bansal
- Jianfeng Gao
- Quoc V. Le
- Sergey Levine
- Percy Liang
- Trevor Darrell
- Dacheng Tao
In Multimodal Learning
- Trevor Darrell
- Marcus Rohrbach
- Devi Parikh
- Mohit Bansal
- Andrew Zisserman
- Qi Wu
- Zhe Gan
- Kristen Grauman
Citations in the corpus
- Self-Supervised Learning of Video-Induced Visual Invariances (2019, background): “Multi-modal learning methods that combine vision and audio [42, 9, 46, 3], and vision and text [57] achieve better performance than unimodal baselines.”
- Fine-Grained Image Classification via Combining Vision and Language (2017, methodology): “Ngiam et al. [26] propose a multi-modal deep learning (MDL) method to combine the audio and video into an autoencoder, which improves the speech signal classification for noisy inputs as well as learns a shared representation across modalities.”
- Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models (2014, methodology): “Other approaches to multimodal learning include the use of deep Boltzmann machines [16], log-bilinear neural language models [2], autoencoders [17], recurrent neural networks [7] and topic-models [18].”
- Deep Fragment Embeddings for Bidirectional Image Sentence Mapping (2014, background): “[20] described an autoencoder that learns audio-video representations through a shared bottleneck layer.”
- Grounded Compositional Semantics for Finding and Describing Images with Sentences (2014, background): “Multimodal embedding methods project data from multiple sources such as sound and video (Ngiam et al., 2011) or images and text.”
- Zero-Shot Learning Through Cross-Modal Transfer (2013, background): “Multimodal embeddings relate information from multiple sources such as sound and video [25] or images and text.”
- Perceiver IO: A General Architecture for Structured Inputs & Outputs (2021, methodology): “This problem has been previously studied using techniques such as Restricted Boltzmann Machines [58], but on much more stereotyped and smaller scale data.”
- Generative Adversarial Text to Image Synthesis (2016, background): “…this small bird has a pink breast and crown, and black primaries and secondaries. the flower has petals that are bright pinkish purple with white stigma this magnificent fellow is almost all black with a red crest, and white cheek patch. this white and yellow flower have thin white petals and a…”
- Coupled Generative Adversarial Networks (2016, methodology): “This is why [3] requires samples from the joint distribution for learning the joint distribution.”
- Multimodal Transformer for Unaligned Multimodal Language Sequences (2019, methodology): “Unlike learning multimodal representations from static domains such as image and textual attributes (Ngiam et al., 2011; Srivastava and Salakhutdinov, 2012), human language contains time-series and thus requires fusing time-varying signals (Liang et al.”
- Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, background): “, via fine-tuning), pre-trained foundation models have achieved strong results when combined and trained together for a number of downstream multimodal (Ngiam et al. 2011) applications including VLMs with LMs for image captioning (e.”
- Learning Deep Structure-Preserving Image-Text Embeddings (2015, background): “Another strand of research on multi-modal embeddings is based on deep learning [3, 24, 25, 31, 35, 44], utilizing such techniques as deep Boltzmann machines [44], autoencoders [35], LSTMs [8], and recurrent neural networks [31, 45].”
- Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding (2016, background): “…et al., 2004) e.g. (Gong et al., 2014; Klein et al., 2015; Plummer et al., 2015), linear models with ranking loss (Frome et al., 2013; Karpathy and Fei-Fei, 2015; Socher et al., 2014; Weston et al., 2011) or non-linear deep learning models (Kiros et al., 2014; Mao et al., 2015; Ngiam et al., 2011).”
- Learning Sight from Sound: Ambient Sound Provides Supervision for Visual Learning (2017, neutral): “Also, in contrast to other recent work in multi-modal representation learning (Ngiam et al 2011; Srivastava and Salakhut-dinov 2012; Andrew et al 2013), our technical approach is based on solving a self-supervised classification problem (rather than fitting a generative model or autoencoder), and our…”
- Visually Indicated Sounds (2015, background): “47,46]. Unlike in these approaches, we learn to predict one sensory modality (sound) from another (vision). There has also been work that trains neural networks from multiple modalities. For example, [34] learned a joint model of audio and video. However, while they study speech using an autoencoder”
- Attribute2Image: Conditional Image Generation from Visual Attributes (2015, background): “Our work can be considered in the context of multimodal learning (e.g., [29, 35, 34]).”
- Learning Transferable Features with Deep Adaptation Networks (2015, background): “Hence, deep neural networks have been explored for domain adaptation (Glorot et al., 2011; Chen et al., 2012), multimodal and multi-source learning problems (Ngiam et al., 2011; Ge et al., 2013), where significant performance gains have been witnessed.”
- SoundNet: Learning Sound Representations from Unlabeled Video (2016, background): “Cross-Modal Learning and Unlabeled Video: Our approach is broadly inspired by efforts to model cross-modal relations [24, 14, 7, 26] and works that leverage large amounts of unlabeled video [25, 41, 8, 40, 39].”
- Learning Deep Representations of Fine-Grained Visual Descriptions (2016, positive): “In [31], audio and video signals were combined in an autoencoder framework, yielding improved speech signal classification for noisy inputs, and learning a shared representation across modalities.”