Towards Domain-Agnostic Contrastive Learning

By Vikas Verma, Minh-Thang Luong, Kenji Kawaguchi, Hieu Pham, Quoc V. Le. 2020

133 citations recorded by the source corpus.

Despite recent success, most contrastive self-supervised learning methods are domain-specific, relying heavily on data augmentation techniques that require knowledge about a particular domain, such as image cropping and rotation. To overcome such limitation, we propose a novel domain-agnostic approach to contrastive learning, named DACL, that is applicable to domains where invariances, and thus, data augmentation techniques, are not readily available. Key to our approach is the use of Mixup noise to create similar and dissimilar examples by mixing data samples differently either at the input or hidden-state levels. To demonstrate the effectiveness of DACL, we conduct experiments across various domains such as tabular data, images, and graphs. Our results show that DACL not only outperforms other domain-agnostic noising methods, such as Gaussian-noise, but also combines well with domain-specific methods, such as SimCLR, to improve self-supervised visual representation learning. Finally, we theoretically analyze our method and show advantages over the Gaussian-noise based contrastive learning approach.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Ocean Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Honglak Lee
2. Jinwoo Shin
3. Philip S. Yu
4. Kibok Lee
5. Kihyuk Sohn
6. Shirui Pan
7. Chuan Zhou
8. Chenliang Li

In Computer Vision and Pattern Recognition

1. Jinwoo Shin
2. Dacheng Tao
3. Philip S. Yu
4. Honglak Lee
5. Kihyuk Sohn
6. Luc Van Gool
7. Kibok Lee
8. Trevor Darrell

In Ocean Engineering

1. Philip S. Yu
2. Kihyuk Sohn
3. Chun-Liang Li

Citations in the corpus

• i-Mix: A Domain-Agnostic Strategy for Contrastive Representation Learning (2021, methodology): “This method can be viewed as introducing structured noises driven by auxiliary data to the principal data with the largest mixing coefficient λ , and the label of the principal data is assigned to the mixed data (Shen et al., 2020; Verma et al., 2020; Zhou et al., 2020).”
• Graph Self-Supervised Learning: A Survey (2021, methodology): “Apart from those methods relying on carefully-crafted negative samples, approaches like BGRL [84] propose to contrast on graph instances themselves and thus alleviate TABLE 3 Main Characteristics of Contrast-Based Graph SSL Approaches Approach Pretext Task Category Downstream Task Level Training Sch”
• Decoupled Mixup for Data-efficient Learning (2022, unknown): “The augmentation policies often rely on particular domain knowledge [43] in different fields.”
• Knowledge Graph Contrastive Learning for Recommendation (2022, methodology): “, image analysis [33] and machine translation [27], we propose to bridge the knowledge graph embedding and the contrastive learning paradigm with auxiliary self-supervised signals.”