FreeLB: Enhanced Adversarial Training for Language Understanding

By Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, Jingjing Liu. 2019

82 citations recorded by the source corpus.

Subfields: Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, neutral): “, 2019c] [Zhu et al., 2019] [Yang et al.”
Adversarial Training for Large Neural Language Models (2020, methodology): “Namely, x is the sub-word embedding in f ( x ; θ ) (Jiang et al., 2019; Zhu et al., 2019).”
Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, methodology): “To power efficient large-scale training, we adopt the recently proposed “free” adversarial training strategy [50, 73, 76], which obtains the gradients of parameters with almost no extra cost when computing the gradients of inputs.”
Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping (2020, background): “Despite its simplicity and ubiquity in modern NLP, this process has been shown to be brittle (Devlin et al., 2019; Phang et al., 2018; Zhu et al., 2019; Raffe et al., 2019), where ﬁne-tuning performance can vary substantially across different training episodes, even with ﬁxed hyperparameter values.”
A Primer in BERTology: What We Know About How BERT Works (2020, background): “• Adversarial token perturbations improve robustness of the model (Zhu et al., 2019).”
SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization (2019, methodology): “Comparing with the current state-of-the-art adversarial training model, FreeLB, SMART outperforms it on 6 GLUE tasks out of a total of 8 tasks (MNLI, RTE, QNLI, MRPC, SST-2 and STS-B) showing the effectiveness of our model.”
CoDA: Contrast-enhanced and Diversity-promoting Data Augmentation for Natural Language Understanding (2020, methodology): “Adversarial Training In another line, adversarial training methods are applied to text data (Zhu et al., 2020; Jiang et al., 2020; Cheng et al., 2020; Aghajanyan et al., 2020) for improving model’s robustness.”
Playing Lottery Tickets with Vision and Language (2021, background): “, image regional features and word embeddings), recent works [81, 30, 20] have shown that adversarial training (AT) can be used as an effective regularization to improve model performance.”
The Microsoft Toolkit of Multi-Task Deep Neural Networks for Natural Language Understanding (2020, methodology): “Finally, to further improve the model robustness, MT-DNN also offers a recipe to apply adversarial training (Madry et al., 2017; Zhu et al., 2019; Jiang et al., 2019) in the ﬁne-tuning stage.”
Adversarial Training for Commonsense Inference (2020, methodology): “…et al., 2019; Miyato et al., 2018), the approaches to estimate δ can be divided into two categories: adversarial training that uses the label y ( Zhu et al., 2020) and adversarial training that uses the model prediction f ( x ; θ ) , i.e. a ”virtual” label (Miyato et al., 2018; Jiang et al.,…”
Boosting Adversarial Training with Hypersphere Embedding (2020, background): “Among the existing defenses, adversarial training (AT) is a general strategy achieving the state-of-the-art robustness under different settings [53, 63, 71, 73, 74, 76, 83].”
Exploring the Vulnerability of Deep Neural Networks: A Study of Parameter Corruption (2020, background): “Related Work Vulnerability of Deep Neural Networks Existing studies concerning vulnerability or robustness of neural networks mostly focus on generating adversarial examples (Goodfellow, Shlens, and Szegedy 2015) and adversarial training algorithms given adversarial examples in the input data (Zhu e”
QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering (2021, unknown): “1 RoBERTa+FreeLB (Zhu et al., 2020) (ensemble) 73.”
MaxUp: A Simple Way to Improve Generalization of Neural Network Training (2020, unknown): “Adversarial ideas have also been used to improvement generalization in a series of recent works (e.g., Xie et al., 2019; Zhu et al., 2020).”