InfoBERT: Improving Robustness of Language Models from An Information Theoretic Perspective

By Boxin Wang, Shuohang Wang, Yu Cheng, Zhe Gan, Ruoxi Jia, Bo Li, Jingjing Liu. 2020

135 citations recorded by the source corpus.

Large-scale language models such as BERT have achieved state-of-the-art performance across a wide range of NLP tasks. Recent studies, however, show that such BERT-based models are vulnerable facing the threats of textual adversarial attacks. We aim to address this problem from an information-theoretic perspective, and propose InfoBERT, a novel learning framework for robust fine-tuning of pre-trained language models. InfoBERT contains two mutual-information-based regularizers for model training: (i) an Information Bottleneck regularizer, which suppresses noisy mutual information between the input and the feature representation; and (ii) a Robust Feature regularizer, which increases the mutual information between local robust features and global features. We provide a principled way to theoretically analyze and improve the robustness of representation learning for language models in both standard and adversarial training. Extensive experiments demonstrate that InfoBERT achieves state-of-the-art robust accuracy over several adversarial datasets on Natural Language Inference (NLI) and Question Answering (QA) tasks.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhiyuan Liu
2. Maosong Sun
3. Mohammad Shoeybi
4. Bryan Catanzaro
5. Yuxiong He
6. Samyam Rajbhandari
7. M. Patwary
8. Qun Liu

In Natural Language Processing

1. Zhiyuan Liu
2. Maosong Sun
3. Qun Liu
4. Mohammad Shoeybi
5. M. Patwary
6. Chenglei Si
7. Shaden Smith
8. Yasheng Wang

In Safety, Robustness & Interpretability

1. Maosong Sun
2. Mohammad Shoeybi
3. Yasheng Wang
4. Bryan Catanzaro
5. Zhengyan Zhang
6. Zhiyuan Liu
7. Fanchao Qi
8. M. Patwary

Citations in the corpus

• Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model (2022, methodology): “On ANLI, the supervised baseline we compare to is InfoBERT [68].”
• Better Robustness by More Coverage: Adversarial Training with Mixup Augmentation for Robust Fine-tuning (2020, methodology): “This is the evaluation method adopted in (Tan et al., 2020; Yin et al., 2020; Zou et al., 2020; Ren et al., 2019; Wang et al., 2020b; Zheng et al., 2020; Wang et al., 2020c, inter alia.”