Latent Translation: Crossing Modalities by Bridging Generative Models

By Yingtao Tian, Jesse Engel. 2019

18 citations recorded by the source corpus.

End-to-end optimization has achieved state-of-the-art performance on many specific problems, but there is no straight-forward way to combine pretrained models for new problems. Here, we explore improving modularity by learning a post-hoc interface between two existing models to solve a new task. Specifically, we take inspiration from neural machine translation, and cast the challenging problem of cross-modal domain transfer as unsupervised translation between the latent spaces of pretrained deep generative models. By abstracting away the data representation, we demonstrate that it is possible to transfer across different modalities (e.g., image-to-audio) and even different types of generative models (e.g., VAE-to-GAN). We compare to state-of-the-art techniques and find that a straight-forward variational autoencoder is able to best bridge the two generative models through learning a shared latent space. We can further impose supervised alignment of attributes in both domains with a classifier in the shared latent space. Through qualitative and quantitative evaluations, we demonstrate that locality and semantic alignment are preserved through the transfer process, as indicated by high transfer accuracies and smooth interpolations within a class. Finally, we show this modular structure speeds up training of new interface models by several orders of magnitude by decoupling it from expensive retraining of base generative models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Multimodal Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Multimodal Learning

In Signal Processing

Citations in the corpus

Multimodal Few-Shot Learning with Frozen Language Models (2021, methodology): “Past work has also explored alternative approaches for post-hoc combination of models for different modalities using latent variables [41].”
Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models (2019, background): “Tian and Engel (2019) considers an intermediary embedding space to transform between independent VAEs, applying constraints on the closeness of embeddings and their individual classiﬁcation performance on labels.”