Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models
By Yuge Shi, Siddharth Narayanaswamy, Brooks Paige, Philip H. S. Torr. Neural Information Processing Systems, 2019
367 citations recorded by the source corpus.
Learning generative models that span multiple data modalities, such as vision and language, is often motivated by the desire to learn more useful, generalisable representations that faithfully capture common underlying factors between the modalities. In this work, we characterise successful learning of such models as the fulfilment of four criteria: i) implicit latent decomposition into shared and private subspaces, ii) coherent joint generation over all modalities, iii) coherent cross-generation across individual modalities, and iv) improved model learning for individual modalities through multi-modal integration. Here, we propose a mixture-of-experts multi-modal variational autoencoder (MMVAE) for learning of generative models on different sets of modalities, including a challenging image language dataset, and demonstrate its ability to satisfy all four criteria, both qualitatively and quantitatively.
Subfields: Artificial Intelligence, Cancer Research, Computer Vision and Pattern Recognition
View Paper: https://www.semanticscholar.org/paper/3a37dedf3f1a0064994c19a4791fc5a892d17b05
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Roman Bachmann
- David Mizrahi
- Andrei Atanov
- Amir Zamir
- Luc Van Gool
- Limin Wang
- Andrew Zisserman
- Cordelia Schmid
In Computer Vision and Pattern Recognition
- Roman Bachmann
- David Mizrahi
- Andrei Atanov
- Amir Zamir
- Luc Van Gool
- Jitendra Malik
- Andrew Zisserman
- Cordelia Schmid
Citations in the corpus
- MultiMAE: Multi-modal Multi-task Masked Autoencoders (2022, background): “Related to MultiMAE are several works that perform multi-modal autoencoding [45,56,60,61,71].”
- Disentangling Shared and Private Neural Dynamics with SPIRE: A Latent Modeling Framework for Deep Brain Stimulation (2025, unknown): “MMVAE assumes mixture-of-experts alignment across modalities, but its architecture lacks explicit mechanisms to separate shared versus private contributions, resulting in latent spaces that cannot support partial reconstructions.”