Unsupervised Adversarial Invariance

By Ayush Jaiswal, Yue Wu, Wael AbdAlmageed, Premkumar Natarajan. 2018

52 citations recorded by the source corpus.

Data representations that contain all the information about target variables but are invariant to nuisance factors benefit supervised learning algorithms by preventing them from learning associations between these factors and the targets, thus reducing overfitting. We present a novel unsupervised invariance induction framework for neural networks that learns a split representation of data through competitive training between the prediction task and a reconstruction task coupled with disentanglement, without needing any labeled information about nuisance factors or domain knowledge. We describe an adversarial instantiation of this framework and provide analysis of its working. Our unsupervised model outperforms state-of-the-art methods, which are supervised, at inducing invariance to inherent nuisance factors, effectively using synthetic data augmentation to learn invariance, and domain adaptation. Our method can be applied to any prediction task, eg., binary/multi-class classification or regression, without loss of generality.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Safety, Robustness & Interpretability

Citations in the corpus

A Survey on Bias and Fairness in Machine Learning (2019, background): “nto the lower bound of their VAE architecture satisfies their proposed model for having the Variational Fair Autoencoder. Similar work, but not targeting fairness specifically, has been introduced in [62]. In [4] authors also propose a debiased VAE architecture called DB-VAE which learns sensitive l”
Invariant Representations through Adversarial Forgetting (2019, methodology): “These methods perform better than data augmentation, due to their approach of invariance by exclusion rather than inclusion (Jaiswal et al. 2018; Jaiswal et al. 2019b; Hsu, Jaiswal, and Natarajan 2019).”
Negative Data Augmentation (2021, background): “They allow for the incorporation of prior knowledge (inductive bias) about properties of typical samples from the underlying data distribution (Jaiswal et al., 2018; Antoniou et al., 2017), e.g., by leveraging invariances to produce additional “positive” examples of how a task should be solved.”
Deep Spectral Clustering Using Dual Autoencoder Network (2019, background): “Models trained in this fashion become robust by exclusion rather than inclusion, and are expected to perform well on clustering tasks, where even the inputs contain unseen nuisance [15].”
Factorizing Knowledge in Neural Networks (2022, background): “Therefore, a recent line of research argues the importance of ﬁnding disentangled variables in representation learning [4,49,45,36,63,13] while providing invariance in learning [14,1,23].”
A Geometric Solution to Fair Representations (2020, methodology): “Similarly, methods that create fair latent features for neural networks (NN) [14, 23] cannot be easily applied to improve fairness in non-NN models.”