Multimodal Machine Learning: A Survey and Taxonomy
By Tadas Baltrusaitis, Chaitanya Ahuja, Louis-philippe Morency. 2017
4,410 citations recorded by the source corpus.
Our experience of the world is multimodal - we see objects, hear sounds, feel texture, smell odors, and taste flavors. Modality refers to the way in which something happens or is experienced and a research problem is characterized as multimodal when it includes multiple such modalities. In order for Artificial Intelligence to make progress in understanding the world around us, it needs to be able to interpret such multimodal signals together. Multimodal machine learning aims to build models that can process and relate information from multiple modalities. It is a vibrant multi-disciplinary field of increasing importance and with extraordinary potential. Instead of focusing on specific multimodal applications, this paper surveys the recent advances in multimodal machine learning itself and presents them in a common taxonomy. We go beyond the typical early and late fusion categorization and identify broader challenges that are faced by multimodal machine learning, namely: representation, translation, alignment, fusion, and co-learning. This new taxonomy will enable researchers to better understand the state of the field and identify directions for future research.
Subfields: Artificial Intelligence, Biomedical Engineering, Multimodal Learning, Evaluation & Benchmarks, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Percy Liang
- Mohit Bansal
- Luke S. Zettlemoyer
- Jianfeng Gao
- Graham Neubig
- Yejin Choi
- Noah A. Smith
- Kyunghyun Cho
In Biomedical Engineering
In Multimodal Learning
- Louis-philippe Morency
- Soujanya Poria
- Percy Liang
- Amir Zadeh
- Devamanyu Hazarika
- Erik Cambria
- Ruslan Salakhutdinov
- Ali Farhadi
In Evaluation & Benchmarks
- Mohit Bansal
- Yejin Choi
- Ivan Vulic
- Douwe Kiela
- Samuel R. Bowman
- Percy Liang
- Rowan Zellers
- Ali Farhadi
In Signal Processing
- Amir Zadeh
- Soujanya Poria
- Louis-philippe Morency
- Percy Liang
- Devamanyu Hazarika
- Erik Cambria
- Rada Mihalcea
- Yang Lu
Citations in the corpus
- MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, background): “[7] Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency.”
- Recent Trends in Deep Learning Based Natural Language Processing (2017, background): “We also expect to see more research on multimodal learning [190] as, in the real world, language is often grounded on (or correlated with) other signals.”
- Supervised Multimodal Bitransformers for Classifying Images and Text (2019, background): “The question of how to effectively combine multimodal information, also known as multimodal fusion, has a long history (Baltruˇsaitis, Ahuja, and Morency 2019).”
- Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCO (2020, background): “We hypothesize that learned representations in multimodal contexts (Baltruaitis et al., 2019) can improve retrieval and discrimination within as well as across modalities.”
- ICON: Interactive Conversational Memory Network for Multimodal Emotion Detection (2018, background): “Current research in this domain is mainly performed from a multimodal learning perspective (Poria et al., 2017a; Baltrušaitis et al., 2018).”
- Network-to-Network Translation with Conditional Invertible Neural Networks (2020, background): “…learning representations for different domains that also allow a domain-to-domain transfer of information between them is significantly more challenging [2]: There is a trade-off between the expressiveness of individual domain representations and their compatibility to another to support transfer.”
- Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities (2018, neutral): “Recently, multimodal sentiment analysis has gained more research interest (Baltrusaitis, Ahuja, and Morency 2017).”
- Foundation models in robotics: Applications, challenges, and the future (2023, neutral): “To deal with noisy 2D bounding boxes from different views, PartSLIP runs a voting and grouping method on super points from 3D, assigns multi-view 2D labels to super points, and finally groups super points to obtain a precise part segmentation.”
- Deep multi-view learning methods: A review (2021, background): “These surveys focus on the problems of specific MVL methods, such as multi-view fusion [10, 27, 28], multi-modal learning [37, 38], multi-view clustering [26] and multi-view representation learning [39, 40].”
- The Missing Ingredient in Zero-Shot Neural Machine Translation (2019, methodology): “Previous work on multi-modal and multi-view representation learning has shown that when given paired data, transferable representations can be much more easily learned by improving some measure of similarity between the alternative views (Baltrušaitis et al., 2018).”
- Knowledge Tracing: A Survey (2022, background): “Fusing signals from multiple feature spaces may enable better representation learning in addition to mitigating noise in data [9].”
- Milvus: A Purpose-Built Vector Data Management System (2021, background): “• Multi-vector query: Each entity is stored as multiple vectors [10].”
- Multimodal sentiment analysis with word-level fusion and reinforcement learning (2017, background): “…multi-modal SVM-MD (Support Vector Machine Multimodal Dictionary) [46] C-MKL (Convolutional Multiple Kernel Learning) [25] is a multimodal RF (Random Forest) is a baseline intended for comparison to a Random is a baseline which always predicts a random sentiment intensity between [ 3 , − 3 ] [46].”
- Multimodal deep learning for biomedical data fusion: a review (2022, background): “Thereby, coordinated representations [11] of each modality could be learned in an unsupervised way, which resulted in encodings of patterns between modalities.”
- Speech gesture generation from the trimodal context of text, audio, and speaker identity (2020, background): “Generally, multimodal data processing includes the representation of each modality, alignment between modalities, and translation between modalities (Baltrušaitis et al. 2018).”
- Factorized Multimodal Transformer for Multimodal Sequential Learning (2019, background): “Learning from multimodal sequential data has been an active, yet challenging research area within the field of machine learning (Baltruˇsaitis et al., 2018).”
- IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages (2022, background): “…models, and evaluating systems that need to jointly reason over multilingual text and multimodal inputs, this way combining two areas of crucial importance: multilingual (Snyder & Barzilay, 2010; Ponti et al., 2019) and multimodal learning (Bernardi et al., 2016; Baltrusaitis et al., 2019).”
- Multi-Task Learning with Deep Neural Networks: A Survey (2020, background): “For a more complete discussion of multi-modal methods, see (Baltrusaitis et al., 2019).”