Controllable Invariance through Adversarial Feature Learning

By Qizhe Xie, Zihang Dai, Yulun Du, Eduard H. Hovy, Graham Neubig. 2017

312 citations recorded by the source corpus.

Learning meaningful representations that maintain the content necessary for a particular task while filtering away detrimental variations is a problem of great interest in machine learning. In this paper, we tackle the problem of learning representations invariant to a specific factor or trait of data. The representation learning process is formulated as an adversarial minimax game. We analyze the optimal equilibrium of such a game and find that it amounts to maximizing the uncertainty of inferring the detrimental factor given the representation while maximizing the certainty of making task-specific predictions. On three benchmark tasks, namely fair and bias-free classification, language-independent generation, and lighting-independent image classification, we show that the proposed framework induces an invariant representation, and leads to better generalization evidenced by the improved performance.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

GAN Inversion: A Survey (2021, background): “Similar to GAN inversion, Adversarial Feature Learning (AFL, also known as BiGAN) [111], [175], [176] or Adversarially Learned Inference (ALI) [112], [177], [178] also aims to learn the inverse mapping of GAN models1.”
Unsupervised Adversarial Invariance (2018, methodology): “An architectural solution to this problem is the development of neural network units that capture specific forms of information, and thus are inherently invariant to certain nuisance factors [3, 19].”
Invariant Representations without Adversarial Training (2018, methodology): “[21] for these post-hoc adversaries, which describes a three-layer”
Adversarial Removal of Demographic Attributes from Text Data (2018, methodology): “…or otherwise) from intermediate representations in order to achieve fair classiﬁcation has been explored by solving an optimization problem (Zemel et al., 2013), as well as by employing adversarial training (Edwards and Storkey, 2015; Louizos et al., 2015; Xie et al., 2017; Zhang et al.,…”
Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection (2020, methodology): “Previous methods are either based on projection on a pre-speciﬁed, user-provided direction (Bolukbasi et al., 2016), or on adding an adversarial objective to an end-to-end training process (Xie et al., 2017).”
Demoting Racial Bias in Hate Speech Detection (2020, methodology): “Adversarial training has been used to remove protected features from word embeddings (Xie et al., 2017; Zhang et al., 2018) and intermediate representations for both texts (Elazar and Goldberg, 2018; Zhang et al.”
Mitigating Information Leakage in Image Representations: A Maximum Entropy Approach (2019, methodology): “A few recent attempts have been made to study related problems, such as learning censored [3], fair [14], or invariant [18] representations of data.”
Information Leakage in Embedding Models (2020, background): “One popular approach is through adversarial training [17, 74] as detailed in Section 7.”
Adversarial Removal of Demographic Attributes Revisited (2019, background): “…Sylvester, 2018; Beutel et al., 2017; Li et al., 2018), as well as representations invariant to domain differences (Ganin and Lempitsky, 2015), clean or noisy speech (Sriram et al., 2018), and invariant to differences between source languages in multi-lingual machine translation (Xie et al., 2017).”
Single-Side Domain Generalization for Face Anti-Spoofing (2020, background): “Therefore, seeking a generalized feature space for the fake faces is difﬁcult to optimize and may also affect the classiﬁcation accuracy for the target domain [1, 40].”
A Survey of Unsupervised Deep Domain Adaptation (2018, background): “ti-source domain adaptation, where there are multiple source domains but still only one target domain. Sun et al. [227] survey multi-source domain adaptation, and since then a number of other methods [28, 88, 95, 157, 184, 192, 256, 259, 281] have been developed for this case. It is also possible to”
Invariant Representations through Adversarial Forgetting (2019, methodology): “The Domain-Adversarial Neural Network (DANN) (Ganin et al. 2016) and the Controllable Ad-versarial Invariance (CAI) model (Xie et al. 2017) indirectly optimize an invariance objective using the gradient-reversal trick (Ganin et al. 2016) that penalizes a DNN if it encodes s .”
Balanced Datasets Are Not Enough: Estimating and Mitigating Gender Bias in Deep Image Representations (2018, methodology): “To mitigate such unlabeled spurious correlations, we adopt an adversarial debiasing approach [34, 2, 38, 6].”
Gender-preserving Debiasing for Pre-trained Word Embeddings (2019, background): “Debiasing can be seen as a problem of hiding information related to a protected attribute such as gender, for which adversarial learning meth-ods (Xie et al., 2017; Elazar and Goldberg, 2018; Li et al., 2018) have been proposed in the fairness-aware machine learning community (Kamiran and Calders,…”
Overlearning Reveals Sensitive Attributes (2019, methodology): “We show that if these attributes are censored (Xie et al., 2017; Moyer et al., 2018), the censored models either fail to learn their speciﬁed tasks, or still leak sensitive information.”
Evading the Adversary in Invariant Representation (2018, methodology): “2013[21], which is also the source for the pre-processing in our baselines [14, 20].”
Benchmarking Bias Mitigation Algorithms in Representation Learning through Fairness Metrics (2021, background): “Adversarial bias-mitigation techniques can be divided 104 into approaches which: (i) seek to mitigate bias through adversarial training directly applied on the 105 class labels, where the class label is an indicator of eligibility [9]; (ii) focus on mitigating bias through 106 enforcing group fairne”
On the Global Optima of Kernelized Adversarial Representation Learning (2019, methodology): “VFAE [17] 57 85 ML-ARL [26] 57 89 MaxEnt-ARL [23] 40 89 -”