Learning a Nonlinear Embedding by Preserving Class Neighbourhood Structure

By Ruslan Salakhutdinov, Geoffrey E. Hinton. 2007

577 citations recorded by the source corpus.

Subfields: Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

Citations in the corpus

Matching Networks for One Shot Learning (2016, background): “The most relevant work is Neighborhood Component Analysis (NCA) [18], and the follow up non-linear version [20].”
Grafit: Learning fine-grained image representations with coarse labels (2020, methodology): “Therefore, inspired by the neighborhood component analysis [23, 39, 47], Wu et al.”
Prototypical Networks for Few-shot Learning (2017, methodology): “A key distinction between our approach and non-linear NCA is that we form a softmax directly over classes, rather than individual points, computed from distances to each class’s prototype representation.”
Deep, Big, Simple Neural Nets for Handwritten Digit Recognition (2010, background): “Training them is hard as back-propagated gradients quickly van-ish exponentially in the number of layers (Hochreiter, 1991; Hochreiter et al., 2001; Hinton, 2007), just like in the ﬁrst recurrent neural networks (Hochreiter & Schmidhu-ber, 1997).”
Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, background): “For example, Salakhutdinov and Hinton [264] proposed to learn a nonlinear transformation, using a deep”
A practical tutorial on autoencoders for nonlinear feature fusion: Taxonomy, models, software and guidelines (2018, background): “Some adversarial AEs, as well as AEs preserving class neighborhood structure [96], can be sorted into this category, since they are able to make use of the class information.”
This looks like that: deep learning for interpretable image recognition (2018, methodology): “nal ﬁlters as visualizable prototypical image parts and also necessitates a novel training procedure. 2 Our work relates closely to other case-based classiﬁcation techniques using k-nearest neighbors [47, 35, 32] or prototypes [33, 2, 48], and very closely, to the Bayesian Case Model [18]. It relate”
Supervised Contrastive Learning (2020, methodology): “Resembling our supervised contrastive approach is the soft-nearest neighbors loss introduced in [41] and used in [53].”
Discovering Hidden Factors of Variation in Deep Networks (2014, background): “Earlier work by [11] demonstrated class-irrelevant aspects in MNIST (style) can be learned by including additional unsupervised units alongside supervised ones in an autoencoder.”
Few-Shot Learning Through an Information Retrieval Lens (2017, background): “NCA and LMNN have also been extended to their deep variants [11] and [12], respectively.”
Improving Generalization via Scalable Neighborhood Component Analysis (2018, methodology): “In the original version [32], the features for the entire dataset needs to be computed at every step of the optimization, making it computationally expensive and not scalable for large datasets.”
A Survey on Metric Learning for Feature Vectors and Structured Data (2013, background): “Source code available at: http://www.ics.uci.edu/~fowlkes/software/nca/ which can be seen as attempting to collapse each class to a single point.18 Unlike NCA, the optimization is done with respect to the matrix M and the problem is thus convex.”
Metric Learning with Adaptive Density Discrimination (2015, background): “…loss for a pair of triplets “symmetrized” for the two positive examples: Neighbourhood Components Analysis Neighbourhood Components Analysis (NCA) and its extensions (Goldberger et al., 2004; Salakhutdinov & Hinton, 2007; Min et al., 2010) have been designed in a similar spirit to Magnet Loss.”
Nonparametric guidance of autoencoder representations using label information (2012, background): “Salakhutdinov and Hinton (2007) combined autoencoder training with neighborhood component analysis (Goldberger et al., 2004), which encouraged the model to encode similar latent representations for inputs belonging to the same class.”
Entangled Watermarks as a Defense against Model Extraction (2020, methodology): “Introduced by Srivastava and Hinton [35], the SNNL was modified and analyzed by Frosst et al. [19].”