Deep learning via semi-supervised embedding

By Jason Aaron Edward Weston, Frédéric Ratle, Hossein Mobahi, Ronan Collobert. 2008

1,070 citations recorded by the source corpus.

We show how nonlinear embedding algorithms popular for use with shallow semi-supervised learning techniques such as kernel methods can be applied to deep multilayer architectures, either as a regularizer at the output layer, or on each layer of the architecture. This provides a simple alternative to existing approaches to deep learning whilst yielding competitive error rates compared to those methods, and existing shallow semi-supervised techniques.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

FitNets: Hints for Thin Deep Nets (2014, background): “Similarly, semi-supervised embedding (Weston et al., 2008) provides guidance to an intermediate layer to help learn very deep networks.”
Deeply-Supervised Nets (2014, background): “There are a few other relevant examples: in [26], they introduce an additional term only at the output layer (to use the discriminative label information to assist with learning the generative model), in strong distinction to our approach of deep supervision in a supervised learning setting; in [30]”
Graph Attention Networks (2017, methodology): “This includes label propagation (LP) (Zhu et al., 2003), semi-supervised embedding (SemiEmb) (Weston et al., 2012), manifold regularization (ManiReg) (Belkin et al., 2006), skip-gram based graph embeddings (Deep-Walk) (Perozzi et al., 2014), the iterative classiﬁcation algorithm (ICA) (Lu & Getoor, ”
Unsupervised Data Augmentation for Consistency Training (2019, methodology): “On this direction, semisupervised learning [4] is one of the most promising methods and recent works can be grouped into three categories: (1) graph-based label propagation via graph convolution [31] and graph embeddings [62], (2) modeling prediction target as latent variables [30], and (3) consiste”
Self-Training With Noisy Student Improves ImageNet Classification (2019, methodology): “Finally, frameworks in semi-supervised learning also include graph-based methods [84, 73, 77, 33], methods that make use of latent variables as target variables [32, 42, 78] and methods based on low-density separation [21, 58, 15], which might provide complementary benefits to our method.”
Revisiting Semi-Supervised Learning with Graph Embeddings (2016, methodology): “Graph-based semi-supervised learning deﬁnes the loss function as a weighted sum of the supervised loss over labeled instances and a graph Laplacian regularization term (Zhu et al., 2003; Zhou et al., 2004; Belkin et al., 2006; Weston et al., 2012).”
Geometric Deep Learning on Graphs and Manifolds Using Mixture Model CNNs (2016, neutral): “ManiReg [4] 59.5% 70.7% SemiEmb [58] 59.0% 71.1% LP [63] 68.0% 63.0% DeepWalk [36] 67.2% 65.3% Planetoid [60] 75.7% 77.2% DCNN [2] 76.80 ± 0.60% 73.00 ± 0.52% GCN [26] 81.59 ± 0.42% 78.72 ± 0.25% MoNet 81.69 ± 0.48% 78.81 ± 0.44% in all the poses were used for training (800 shapes in total); the…”
Unsupervised and Semi-supervised Learning with Categorical Generative Adversarial Networks (2015, methodology): “We either used the full set (Weston et al., 2012) 7.75 - SWWAE (Zhao et al., 2015) 8.71 ± 0 .”
Natural Language Processing (Almost) from Scratch (2011, background): “Note that our procedure is clearly linked to the (semi-supervised) deep learning procedures of (Hinton et al., 2006; Bengio et al., 2007; Weston et al., 2008).”
Semi-supervised Learning with Deep Generative Models (2014, background): “by training feed-forward classifiers with an additional penalty from an auto-encoder or other unsupervised embedding of the data (Ranzato and Szummer, 2008; Weston et al., 2012).”
Representation Learning: A Review and New Perspectives (2012, methodology): “representations, like semi-supervised embedding [217].”
Semi-supervised Learning with Ladder Networks (2015, methodology): “Test error without data augmentation % with # of used labels 100 all EmbedCNN (Weston et al. , 2012) 7.75 SWWAE (Zhao et al. , 2015) 9.17 0.71 Baseline: Conv-Small, supervised only 6.43 ( ± 0.84) 0.36 Conv-FC 0.91 ( ± 0.14) Conv-Small, Γ -model 0.86 ( 0.41) Table 2: CNN results for MNIST”
Stacked What-Where Auto-encoders (2015, background): “For instance, a joint embedding learning approach is proposed [29].”
Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning (2016, background): “We show through numerous experiments that this combination leads to a competitive semi-supervised learning method.”
Learning with Pseudo-Ensembles (2014, methodology): “From left-to-right the methods are Transductive SVM , neural net, convolutional neural net, EmbedNN [23], Manifold Tangent Classifier [16], Pseudo-Label [14], standard dropout plus fuzzing [9], dropout plus fuzzing with pre-training, PEA, and PEA with pre-training.”
Distributed learning of deep neural network over multiple agents (2018, methodology): “Additionally, we can combine the losses of generative and predictive segments to perform semi supervised learning, adding a regularization component while training on fewer samples [33].”
Attacking Graph-based Classification via Manipulating the Graph Structure (2019, methodology): “Some graph neural network methods [5, 38, 52, 69] learn the node feature vectors and classify nodes simultaneously.”