Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities

By Hai Pham, Percy Liang, Thomas Manzini, Louis-philippe Morency, Barnab'as P'oczos. 2018

560 citations recorded by the source corpus.

Multimodal sentiment analysis is a core research area that studies speaker sentiment expressed from the language, visual, and acoustic modalities. The central challenge in multimodal learning involves inferring joint representations that can process and relate information from these modalities. However, existing work learns joint representations by requiring all modalities as input and as a result, the learned representations may be sensitive to noisy or missing modalities at test time. With the recent success of sequence to sequence (Seq2Seq) models in machine translation, there is an opportunity to explore new ways of learning joint representations that may not require all input modalities at test time. In this paper, we propose a method to learn robust joint representations by translating between modalities. Our method is based on the key insight that translation from a source to a target modality provides a method of learning joint representations using only the source modality as input. We augment modality translations with a cycle consistency loss to ensure that our joint representations retain maximal information from all modalities. Once our translation model is trained with paired multimodal data, we only need data from the source modality at test time for final sentiment prediction. This ensures that our model remains robust from perturbations or missing information in the other modalities. We train our model with a coupled translationprediction objective and it achieves new state-of-the-art results on multimodal sentiment analysis datasets: CMU-MOSI, ICTMMMO, and YouTube. Additional experiments show that our model learns increasingly discriminative joint representations with more input modalities while maintaining robustness to missing or perturbed modalities.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Multimodal Transformer for Unaligned Multimodal Language Sequences (2019, methodology): “…models, as well as Recurrent Attended Variation Embedding Network (RAVEN) (Wang et al., 2019) and Multimodal Cyclic Translation Network (MCTN) (Pham et al., 2019), that achieved SOTA results on various word-aligned human 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 multimodal language…”
Does language help generalization in vision models? (2021, background): “Learning vision models using language supervision has gained popularity recently [18, 24, 4, 10, 17, 1, 8, 21].”
MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis (2020, background): “• MulT [52]proposesamultimodaltransformer architecturewhich • MCTN [39]implementsatranslation-based modelwithanencoder-decoder • ARGF [30] is a recent model that learns a common embedding • BC-LSTM [43] is a contextual-utterance model which learns an • CH-FUSION [32] is a strong baseline which…”
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction (2022, unknown): “Owens & Efros (2018) learns a multimodal network to predict whether the audio and visual streams of a video are temporally synchronized while Pham et al. (2019) applies a cyclic translation between different modalities.”
Integrating Multimodal Information in Large Pretrained Transformers (2020, background): “Works in this area often focus on novel multimodal neural architectures (Pham et al., 2019; Hazarika et al., 2018) and multimodal fusion approaches (Liang et al.”
Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities (2021, methodology): “Additionally, the sequential translation-based models require translation and generation of videos, audios, and text, which are difﬁcult to train especially with limited training samples (Li et al., 2018; Pham et al., 2019).”
SMIL: Multimodal Learning with Severely Missing Modality (2021, background): “Although there exist a bunch of research efforts (Tsai et al. 2019; Pham et al. 2019) in developing novel methods to tackle the incompleteness of testing data, few of them can handle incomplete training modalities.”
Think Locally, Act Globally: Federated Learning with Local and Global Representations (2020, background): “Representation Learning involves learning informative features from data that are useful for generative [69, 71, 68] and discriminative [48, 47, 74, 58] tasks.”