On Deep Multi-View Representation Learning

By Weiran Wang, Raman Arora, Karen Livescu, J. Bilmes. 2015

1,043 citations recorded by the source corpus.

We consider learning representations (features) in the setting in which we have access to multiple unlabeled views of the data for representation learning while only one view is available at test time. Previous work on this problem has proposed several techniques based on deep neural networks, typically involving either autoencoder-like networks with a reconstruction objective or paired feedforward networks with a correlation-based objective. We analyze several techniques based on prior work, as well as new variants, and compare them experimentally on visual, speech, and language domains. To our knowledge this is the first head-to-head comparison of a variety of such techniques on multiple tasks. We find an advantage for correlation-based representation learning, while the best results on most tasks are obtained with our new variant, deep canonically correlated autoencoders (DCCAE).

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Representations by Maximizing Mutual Information Across Views (2019, methodology): “More recent work based on multiview CCA has extended these approaches beyond formally tractable settings, and is more akin to our own work [Wang et al., 2015].”
Domain Adaptation for Visual Applications: A Comprehensive Survey (2017, methodology): “ain Adaptation for Visual Applications: A Comprehensive Survey 35 As such, multi-view learning methods are related to HDA/HTL as discussed also in Section 3.3. Nevertheless, while multi-view learning [90, 91] assumes that multi-view examples are available during training, in the case of HDA [116, 11”
Deep multi-view learning methods: A review (2021, methodology): “For the deep multi-view object recognition methods, we compare the classification accuracy (Accuracy) [49] of the following 660 approaches: multi-view CNN (MVCNN) [49], Kd-tree based network (KD-Net) [51], multi-view harmonized bilinear network (MHBN) [52], group-view CNN (GVCNN) [50] and dynamic ro”
A Cookbook of Self-Supervised Learning (2023, background): “, 2017], and deep canonically correlated autoencoders [Wang et al., 2015].”
A Survey of Multi-View Representation Learning (2016, neutral): “In particular, DCCAE consists of two autoencoders and optimizes the combination of canonical correlation between the learned bottleneck representations and the reconstruction errors of the autoencoders.”
Multi-view learning overview: Recent progress and new challenges (2017, background): “So far, multi-view deep representation learning has two main strategies [130].”
Learning from Multiview Correlations in Open-domain Videos (2018, background): “Reconstruction-based and correlation-based learning can be complementary, as described in [18], which examines both losses on an audio-visual speech recognition dataset.”
Contextual Learning (2020, neutral): “In a recent article, Wang et al. (2015) suggest and compare deep architectures that combine multi-task and multi-view learning, and show that a deep canonically correlated auto-encoder gives superior results for visual, speech, and language learning.”
Learning Distributed Representations of Symbolic Structure Using Binding and Unbinding Operations (2018, background): “, 2013); even samples from different modalities can be easily compared in a common representation space (de Sa, 1993; Wang et al., 2015).”
Zero-Shot Recognition via Structured Prediction (2016, background): “There also exist other approaches such as transfer learning [35], multimodel learning [36], multi-view learning [37], multi-domain and multi-task learning [38], that have been applied to zero-shot learning [38].”
Extending Long Short-Term Memory for Multi-View Structured Learning (2016, neutral): “[23] have compared several deep multi-view representation learning models and proposed a new variant combining Canonical Correlation Analysis (CCA) and an Autoencoder.”
Quantifying the Visual Concreteness of Words and Topics in Multimodal Datasets (2018, methodology): “, 2013) (DCCA), a method that is competitive with traditional kernel CCA (Wang et al., 2015a) but less memory-intensive to train.”
Multi-View Clustering via Deep Matrix Factorization (2017, methodology): “One major difference is that (Andrew et al. 2013; Wang et al. 2015) are based on Canonical Correlation Analysis (CCA), which is limited to 2-view case, while our method has no such limitation.”
Deep Multimodal Representation Learning: A Survey (2019, background): “[118] proposed a variant of DACC name deep canonically correlated autoencoders (DCCAE).”
COMIC: Multi-view Clustering Without Parameter Selection (2019, background): “We compare our methods with nine MvC approaches, namely, the vanilla k-means, the normalized spectral clustering (SC) (Ng et al., 2002), low rank representation (LRR) (Liu et al., 2016), diversity-induced multi-view subspace clustering (DiMSC) (Cao et al., 2015), latent multi-view subspace clusterin”
Multi-view Response Selection for Human-Computer Conversation (2016, background): “In 2015, a further study conducted by Wang et al. (2015a) achieved better results using tree structures as the input of a DNN model.”