Invariant Representations without Adversarial Training

By Daniel Moyer, Shuyang Gao, Rob Brekelmans, Aram Galstyan, Greg Ver Steeg. 2018

250 citations recorded by the source corpus.

Representations of data that are invariant to changes in specified factors are useful for a wide range of problems: removing potential biases in prediction problems, controlling the effects of covariates, and disentangling meaningful factors of variation. Unfortunately, learning representations that exhibit invariance to arbitrary nuisance factors yet remain useful for other tasks is challenging. Existing approaches cast the trade-off between task performance and invariance in an adversarial way, using an iterative minimax optimization. We show that adversarial training is unnecessary and sometimes counter-productive; we instead cast invariant representation learning as a single information-theoretic objective that can be directly optimized. We demonstrate that this approach matches or exceeds performance of state-of-the-art adversarial approaches for learning fair representations and for generative modeling with controllable transformations.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Safety, Robustness & Interpretability

Citations in the corpus

Unsupervised Speech Representation Learning Using WaveNet Autoencoders (2019, background): “since this information is explicitly made available to the decoder conditioning network, and therefore need not be preserved in the latent encoding [54].”
Attributing Fair Decisions with Attention Interventions (2021, methodology): “…for fair machine learning include applying complex transformations to the data so that resulting representations are fair (Gupta et al. 2021; Moyer et al. 2018; Roy and Boddeti 2019; Jaiswal et al. 2020; Song et al. 2019), adding regularizers to incorporate fairness (Zafar et al. 2017;…”
Man is to Person as Woman is to Location: Measuring Gender Bias in Named Entity Recognition (2019, background): “From representation learning (Moyer et al. 2018) to graph embedding (Bose and Hamilton 2019) to community detection (Mehrabi et al. 2019a) and clustering (Backurs et al. 2019), researchers have studied biases in these areas and tried to address them by pointing out the observed problems and…”
Flexibly Fair Representation Learning by Disentanglement (2019, background): “Several recent VAE variants have incorporated label information into latent vari-Flexibly able learning for image synthesis (Klys et al., 2018) and single-attribute fair representation learning (Song et al., 2019; Botros & Tomczak, 2018; Moyer et al., 2018).”
Learning Robust Representations by Projecting Superficial Statistics Out (2018, background): “Several heuristics have been proposed to force a network to “forget” some part of a representation, such as adversarial training (Ganin et al., 2016) or information-theoretic regularization (Moyer et al., 2018).”
On Variational Bounds of Mutual Information (2019, background): “These techniques have proven useful in a variety of domains, from restricting the capacity of discriminators in GANs (Peng et al., 2018) to preventing representations from containing information about protected attributes (Moyer et al., 2018).”
Information Leakage in Embedding Models (2020, background): “is through directly minimizing the mutual information between the sensitive attributes and the deep representations [47, 50].”
Overlearning Reveals Sensitive Attributes (2019, methodology): “The requirement that z not reveal s can be formalized as an independence constraint z ⊥ s, but independence is intractable to measure in practice, thus the requirement is relaxed to a constraint on the mutual information between z and s [28, 30].”
Controlling Text Edition by Changing Answers of Specific Questions (2021, background): “Implicit disentanglement (Higgins et al., 2017; Chen et al., 2018; Moyer et al., 2018; Mathieu et al., 2018; Kim and Mnih, 2018) separates the latent space into several components in a purely unsupervised way, expecting that each component corresponds to an attribute.”
A Geometric Solution to Fair Representations (2020, methodology): “Similarly, methods that create fair latent features for neural networks (NN) [14, 23] cannot be easily applied to improve fairness in non-NN models.”
On the Global Optima of Kernelized Adversarial Representation Learning (2019, unknown): “No Privacy SSGD ML-ARL[24] MaxEnt-ARL[21] Linear-SARL Kernel-SARL”
Invariant Representations through Adversarial Forgetting (2019, methodology): “…recent years methods have emerged that remove s from the latent representations ˜ z , penalizing the model for the presence of s (Jaiswal et al. 2018; Jaiswal et al. 2019a; Li, Swersky, and Zemel 2014; Lopez et al. 2018; Louizos et al. 2016; Moyer et al. 2018; Xie et al. 2017; Zemel et al. 2013).”
Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principle (2018, background): “Reference [33] introduces IðC;LÞ, where C is some nuisance factor or discriminatory trait, as a”