CLINE: Contrastive Learning with Semantic Negative Examples for Natural Language Understanding

By Dong Wang, Ning Ding, Pijian Li, Haitao Zheng. 2021

133 citations recorded by the source corpus.

Despite pre-trained language models have proven useful for learning high-quality semantic representations, these models are still vulnerable to simple perturbations. Recent works aimed to improve the robustness of pre-trained models mainly focus on adversarial training from perturbed examples with similar semantics, neglecting the utilization of different or even opposite semantics. Different from the image processing field, the text is discrete and few word substitutions can cause significant semantic changes. To study the impact of semantics caused by small perturbations, we conduct a series of pilot experiments and surprisingly find that adversarial training is useless or even harmful for the model to detect these semantic changes. To address this problem, we propose Contrastive Learning with semantIc Negative Examples (CLINE), which constructs semantic negative examples unsupervised to improve the robustness under semantically adversarial attacking. By comparing with similar and opposite semantic examples, the model can effectively perceive the semantic changes caused by small perturbations. Empirical results show that our approach yields substantial improvements on a range of sentiment analysis, reasoning, and reading comprehension tasks. And CLINE also ensures the compactness within the same semantics and separability across different semantics in sentence-level.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhiyuan Liu
2. Minlie Huang
3. Jie Tang
4. Maosong Sun
5. Xu Han
6. Xipeng Qiu
7. Ji-rong Wen
8. Wayne Xin Zhao

In Computer Vision and Pattern Recognition

1. Zhiyuan Liu
2. Ji-rong Wen
3. Yang Liu
4. Xu Han
5. Wayne Xin Zhao
6. Minlie Huang
7. Jun Zhu
8. Jie Tang

Citations in the corpus

• Pre-Trained Models: Past, Present and Future (2021, background): “For example, PTMs can be easily fooled by synonym replacement (Jin et al., 2020; Zang et al., 2020; Wang et al., 2021a).”
• Prompt-Learning for Fine-Grained Entity Typing (2021, methodology): “By applying self-supervised learning on large-scale unlabeled corpora, PLMs can capture rich lexical (Jawahar et al., 2019), syntactic (Hewitt and Manning, 2019; Wang et al., 2021), and factual knowledge (Petroni et al., 2019) that well benefits downstream NLP tasks.”
• AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, methodology): “Models like CLINE [161] are obtained by further pretraining RoBERTa model using generative (MLM), contrastive, and adversarial SSL (RTD).”