Multimodal Generative Models for Scalable Weakly-Supervised Learning

By Mike Wu, Noah D. Goodman. 2018

499 citations recorded by the source corpus.

Multiple modalities often co-occur when describing natural phenomena. Learning a joint representation of these modalities should yield deeper and more useful representations. Previous work have proposed generative models to handle multi-modal input. However, these models either do not learn a joint distribution or require complex additional computations to handle missing data. Here, we introduce a multimodal variational autoencoder that uses a product-of-experts inference network and a sub-sampled training paradigm to solve the multi-modal inference problem. Notably, our model shares parameters to efficiently learn under any combination of missing modalities, thereby enabling weakly-supervised learning. We apply our method on four datasets and show that we match state-of-the-art performance using many fewer parameters. In each case our approach yields strong weakly-supervised results. We then consider a case study of learning image transformations---edge detection, colorization, facial landmark segmentation, etc.---as a set of modalities. We find appealing results across this range of tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Foundation Posteriors for Approximate Probabilistic Inference (2022, positive): “Second, we compute the variance of the log importance weights (IW), as in [68].”
SMIL: Multimodal Learning with Severely Missing Modality (2021, background): “• MVAE (Wu and Goodman 2018) is proposed for multimodal generative task.”
Variational Item Response Theory: Fast, Accurate, and Expressive (2020, background): “If we assume each component qφ(ai|dj , ri,j) is Gaussian, then qφ(ai|d1:M , ri,1:M ) is Gaussian as well, being a Product-OfExperts [22, 45].”
MultiMAE: Multi-modal Multi-task Masked Autoencoders (2022, background): “Related to MultiMAE are several works that perform multi-modal autoencoding [45,56,60,61,71].”
Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Tasks (2019, background): “assumption for many sensor fusion works with Bayesian filters [68] as well as recent works on multimodal generative models [60], [61].”
Lifelong Mixture of Variational Autoencoders (2021, background): “In another direction of research, mixtures of VAEs have been employed for continuous learning [24], [25], [26], [27].”
Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models (2019, methodology): “Moreover Wu and Goodman (2018) noted that P O E does not work well when observations across all modalities are always presented during training, requiring artiﬁcial subsampling of the observations to ensure that the individual modalities are learnt faithfully.”