Towards Principled Unsupervised Learning

By Ilya Sutskever, Rafal Józefowicz, Karol Gregor, Danilo Jimenez Rezende, Timothy P. Lillicrap, Oriol Vinyals. 2015

50 citations recorded by the source corpus.

General unsupervised learning is a long-standing conceptual problem in machine learning. Supervised learning is successful because it can be solved by the minimization of the training error cost function. Unsupervised learning is not as successful, because the unsupervised objective may be unrelated to the supervised task of interest. For an example, density modelling and reconstruction have often been used for unsupervised learning, but they did not produced the sought-after performance gains, because they have no knowledge of the supervised tasks. In this paper, we present an unsupervised cost function which we name the Output Distribution Matching (ODM) cost, which measures a divergence between the distribution of predictions and distributions of labels. The ODM cost is appealing because it is consistent with the supervised cost in the following sense: a perfect supervised classifier is also perfect according to the ODM cost. Therefore, by aggressively optimizing the ODM cost, we are almost guaranteed to improve our supervised performance whenever the space of possible predictions is exponentially large. We demonstrate that the ODM cost works well on number of small and semi-artificial datasets using no (or almost no) labelled training cases. Finally, we show that the ODM cost can be used for one-shot domain adaptation, which allows the model to classify inputs that differ from the input distribution in significant ways without the need for prior exposure to the new domain.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Evaluation & Benchmarks

Citations in the corpus

Improved Techniques for Training GANs (2016, methodology): “Like many deep generative models, GANs have previously been applied to semi-supervised learning [13, 14], and our work can be seen as a continuation and reﬁnement of this effort.”
Semi-Supervised Learning with Generative Adversarial Networks (2016, background): “We are excited to explore the following related ideas: • Share some (but not all) of the weights between D and C, as in the dual autoencoder (Sutskever et al., 2015).”
PixelGAN Autoencoders (2017, background): “Recently several GAN-based methods have been developed to learn a cross-domain mapping between two different domains [31–33, 6, 34].”
One-Sided Unsupervised Domain Mapping (2017, background): “In CoGAN, the two domains are assumed to be similar and their generators (and GAN discriminators) share many of the layers weights, similar to [21].”
A Review on Machine Learning and Deep Learning Perspectives of IDS for IoT: Recent Updates, Security Issues, and Challenges (2020, background): “The neurons are initialized with random weights and are multiplied with input parameters to return an output [65, 66].”
Unsupervised Learning of Predictors from Unpaired Input-Output Samples (2016, background): “A recent study that is more closely related to what we describe in this paper is [27], which proposes the output distribution matching (ODM) as an alternative unsupervised learning objective to the likelihood function of the data.”
Language Models are Unsupervised Multitask Learners (2019, background): “In this slightly toy setting, the concerns with density estimation as a principled training objective discussed in (Sutskever et al., 2015) are side stepped.”
How Generative Adversarial Networks and Its Variants Work : An Overview of GAN (2017, methodology): “We can produce real-like data, which can be used for filling missing data in semi-supervised learning by training a generative model [105],[109].”
Improving the Improved Training of Wasserstein GANs: A Consistency Term and Its Dual Effect (2018, background): “…adversarial networks (GANs) among them (Goodfellow et al., 2014) have the biggest impact thus far on a variety of problems and applications (Radford et al., 2015; Denton et al., 2015; Im et al., 2016; Isola et al., 2016; Springenberg, 2015; Sutskever et al., 2015; Odena, 2016; Zhu et al., 2017).”