Good Semi-supervised Learning That Requires a Bad GAN

By Zihang Dai, Zhilin Yang, Fan Yang, William W. Cohen, Ruslan Salakhutdinov. 2017

507 citations recorded by the source corpus.

Semi-supervised learning methods based on generative adversarial networks (GANs) obtained strong empirical results, but it is not clear 1) how the discriminator benefits from joint training with a generator, and 2) why good semi-supervised classification performance and a good generator cannot be obtained at the same time. Theoretically, we show that given the discriminator objective, good semisupervised learning indeed requires a bad generator, and propose the definition of a preferred generator. Empirically, we derive a novel formulation based on our analysis that substantially improves over feature matching GANs, obtaining state-of-the-art results on multiple benchmark datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Self-Training With Noisy Student Improves ImageNet Classification (2019, methodology): “Finally, frameworks in semi-supervised learning also include graph-based methods [84, 73, 77, 33], methods that make use of latent variables as target variables [32, 42, 78] and methods based on low-density separation [21, 58, 15], which might provide complementary benefits to our method.”
Data-Efficient Image Recognition with Contrastive Predictive Coding (2019, background): “isms which can broadly be grouped into two categories: generative models that learn compact repre1 arXiv:1905.09272v1 [cs.CV] 22 May 2019 sentations that are capable of predicting pixel-level details [34,54,12], and self-supervised methods that formulate alternative prediction tasks in which missing”
Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Sampling (2020, background): “, 2014) are state-of-the-art models for a large variety of tasks such as image generation, semi-supervised learning (Dai et al., 2017), image editing (Yi et al.”
Generative Modeling by Estimating Gradients of the Data Distribution (2019, background): “To list a few, they have been used to generate high-fidelity images [26, 6], synthesize realistic speech and music fragments [58], improve the performance of semi-supervised learning [28, 10], detect adversarial examples and other anomalous data [54], imitation learning [22], and explore promising s”
BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling (2019, neutral): “58 ) SSLGAN [6] 0 .”
FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence (2020, background): “These heavily-augmented examples are almost certainly outside of the data distribution, which has in fact been shown to be potentially beneﬁcial for SSL [11].”
Generative Feature Replay For Class-Incremental Learning (2020, background): “Additionally, the generation of highresolution images does not guarantee that they are able to capture a large enough variety of visual concepts with a good discriminative power [6].”
Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples (2017, methodology): “It is also interesting future directions applying our methods for other related tasks: network calibration (Guo et al., 2017), Bayesian probabilistic models (Li & Gal, 2017; Louizos & Welling, 2017), ensemble method (Lakshminarayanan et al., 2016) and semi-supervised learning (Dai et al., 2017).”
Machine Learning with Membership Privacy using Adversarial Regularization (2018, methodology): “Adversarial training has also been used for regularizing, hence generalizing, a model [11, 14, 29, 35, 36, 38].”
Maximum Likelihood Training of Score-Based Diffusion Models (2021, background): “Since better likelihoods are useful for applications including compression [21, 20, 51], semi-supervised learning [10], adversarial purification [47], and comparing against likelihood-based generative models, we seek a training objective for score-based diffusion models that is as efficient as score”
A Baseline for Few-Shot Image Classification (2019, methodology): “Modern augmentation techniques such as [32, 33, 34] or graph-based approaches [35] can also be used with our approach; we used the entropic penalty for the sake of simplicity.”
Smooth Neighbors on Teacher Graphs for Semi-Supervised Learning (2017, methodology): “For example, “fake” samples can be generated in low density regions where the training data is rare [40, 15] based on the low density separation assumption.”
FreeMatch: Self-adaptive Thresholding for Semi-supervised Learning (2022, background): “To alleviate the reliance on labeled data, semi-supervised learning (SSL) [4, 5, 6, 7, 8, 9, 10] is developed to improve the model’s generalization performance by exploiting a massive amount of unlabeled data.”
Semi-supervised Learning on Graphs with Generative Adversarial Nets (2018, background): “Analysis on the trade-off between the classification performance of semi-supervised and the quality of generator was given in [8].”
Adversarial Reciprocal Points Learning for Open Set Recognition (2021, methodology): “Compared with [22], [28], [37], [38], the main differences of the proposed instantiated adversarial enhancement are as follows: First, the proposed method uses the adversarial mechanism between the close space Sk and the global open space OG formed by the reciprocal points.”
How good is my GAN? (2018, methodology): “Indeed, GANs have been used to train classification networks in a semi-supervised fashion [13, 52] or to facilitate domain adaptation [10].”