Adversarial Contrastive Estimation

By Avishek Joey Bose, Huan Ling, Yanshuai Cao. 2018

60 citations recorded by the source corpus.

Learning by contrasting positive and negative samples is a general strategy adopted by many methods. Noise contrastive estimation (NCE) for word embeddings and translating embeddings for knowledge graphs are examples in NLP employing this approach. In this work, we view contrastive learning as an abstraction of all such methods and augment the negative sampler into a mixture distribution containing an adversarially learned sampler. The resulting adaptive sampler finds harder negative examples, which forces the main model to learn a better representation of the data. We evaluate our proposal on learning word embeddings, order embeddings and knowledge graph embeddings and observe both faster convergence and improved results on multiple metrics.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

Flow Contrastive Estimation of Energy-Based Models (2019, background): “NCE and it variants has gained popularity in natural language processing (NLP) [23, 57, 2, 4].”
Compositional Fairness Constraints for Graph Embeddings (2019, methodology): “…methods that make use of a corruption distribution such as noise contrastive estimation (Dyer, 2014; Mnih & Teh, 2012) and similar variants (Bose et al., 2018), where the loss over a batch of edges E batch ⊆ E train is given by: e ∈E edge L edge ( s ( e ) , s ( e − 1 ) , ..., s ( e − m ))…”
Deep Graph Infomax (2018, methodology): “…methods is primarily based on sampling of random pairs, with recent work adapting this approach to use a curriculum-based negative sampling scheme (with progressively “closer” negative examples; Ying et al., 2018a) or introducing an adversary to select the negative examples (Bose et al., 2018).”
Negative Data Augmentation (2021, methodology): “Bose et al. (2018) use adversarial training to generate hard negatives that fool the discriminator for NLP tasks whereas we obtain NDA data from positive data to improve image generation and representation learning.”
How to Train Your Energy-Based Models (2021, methodology): “Many methods have been proposed to automatically tune the noise distribution, such as Adversarial Contrastive Estimation (Bose et al., 2018), Conditional NCE (Ceylan and Gutmann, 2018) and Flow Contrastive Estimation (Gao et al., 2020).”
Energy-Based Imitation Learning (2020, background): “Although many recent works [4, 15] depend on an adversarially learned sampler, there are no theoretical guarantees showing that such an alternating learning style is more advanced than other streams of methods.”
Adversarial Attack and Defense on Graph Data: A Survey (2018, background): “adversarial contrastive learning [15] to tackle the vulnerabilities of GNN models to adversarial attacks due to training”
Self-supervised Context-aware Style Representation for Expressive Speech Synthesis (2022, unknown): “Besides, contrastive learning is also widely used in NLP and speech areas [13, 14, 15, 16].”
Seeking Patterns, Not just Memorizing Procedures: Contrastive Learning for Solving Math Word Problems (2021, unknown): “Bose et al. To the best of our knowledge, this is the ﬁrst work to adopt contrastive learning to MWP solving.”