Conditional BERT Contextual Augmentation

By Xing Wu, Shangwen Lv, Liangjun Zang, Jizhong Han, Songlin Hu. 2018

348 citations recorded by the source corpus.

Data augmentation methods are often applied to prevent overfitting and improve generalization of deep neural network models. Recently proposed contextual augmentation augments labeled sentences by randomly replacing words with more varied substitutions predicted by language model. Bidirectional Encoder Representations from Transformers (BERT) demonstrates that a deep bidirectional language model is more powerful than either an unidirectional language model or the shallow concatenation of a forward and backward model. We propose a novel data augmentation method for labeled sentences called conditional BERT contextual augmentation. We retrofit BERT to conditional BERT by introducing a new conditional masked language model (The term “conditional masked language model” appeared once in original BERT paper, which indicates context-conditional, is equivalent to term “masked language model”. In our paper, “conditional masked language model” indicates we apply extra label-conditional constraint to the “masked language model”.) task. The well trained conditional BERT can be applied to enhance contextual augmentation. Experiments on six various different text classification tasks show that our method can be easily applied to both convolutional or recurrent neural networks classifier to obtain improvement.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

TinyBERT: Distilling BERT for Natural Language Understanding (2019, methodology): “Specifically, we use the language model to predict word replacements for single-piece words (Wu et al., 2019), and use the word embeddings to retrieve the most similar words as word replacements for multiple-pieces words3.”
Do Not Have Enough Data? Deep Learning to the Rescue! (2019, methodology): “Previous textual data augmentation approaches focus on sample alteration (Kobayashi 2018; Wu et al. 2019; Wei and Zou 2019; Mueller and Thyagarajan 2016; Jungiewicz and Smywinski-Pohl 2019), in which a single sentence is altered in one way or another, to generate a new sentence while preserving the…”
Data Augmentation using Pre-trained Transformer Models (2020, methodology): “To alleviate this issue, Wu et al. (2019) proposed conditional BERT (CBERT) model which extends BERT (Devlin et al., 2018) masked language modeling (MLM) task by considering class labels to predict the masked tokens.”
Pre-trained models for natural language processing: A survey (2020, background): “Indeed, mining knowledge from PTMs can be more flexible, such as feature extraction, knowledge distillation [203], data augmentation [192, 89], using PTMs as external knowledge [132].”
G-DAug: Generative Data Augmentation for Commonsense Reasoning (2020, methodology): “…using a thesaurus (Zhang et al., 2015; Wei and Zou, 2019) and word embeddings (Wang and Yang, 2015; Fadaee et al., 2017; Kobayashi, 2018; Wu et al., 2019), and recently, generative models for synthesizing novel examples for text classiﬁcation and reading comprehension (Anaby-Tavor et…”
Text Detoxification using Large Pre-trained Neural Models (2021, methodology): “We compare our models with state-of-the-art methods described in Section 2: DRG-TemplateBased, DRG-RetrieveOnly, Mask&Infill, DLSM, STRAP, and SST.”
A Diverse Data Augmentation Strategy for Low-Resource Neural Machine Translation (2020, methodology): “replace the bi-directional model with BERT [24], which is a more powerful model, then use it to generate a set of substitute words [25].”
Soft Contextual Data Augmentation for Neural Machine Translation (2019, methodology): “A recent new method is contextual augmentation (Kobayashi, 2018; Wu et al., 2018), which re-places words with other words that are predicted using language model at the corresponding word position.”
How Effective is Task-Agnostic Data Augmentation for Pretrained Transformers? (2020, methodology): “2.1 Datasets Following Wei and Zou (2019) and Wu et al. (2019) we adopt 4 classiﬁcation datasets on which general data augmentation techniques demonstrated strong performance gains, and include 2 more from the GLUE benchmark (Wang et al., 2018).”
An Analysis of Simple Data Augmentation for Named Entity Recognition (2020, methodology): “In addition, there are two special word replacement cases, inspired by dropout and masked language modeling: replacing a word by a zero word (i.e., dropping entire word embeddings) (Iyyer et al., 2015), or by a [MASK] token (Wu et al., 2018).”
Learning Data Manipulation for Augmentation and Weighting (2019, methodology): “We compare our approach with a variety of previous methods that were designed for specific manipulation schemes: (1) For text data augmentation, we compare with the latest model-based augmentation [49] which uses a fixed conditional BERT language model for word substitution (section 4.”
Data Augmentation Approaches in Natural Language Processing: A Survey (2021, methodology): “[51] fine-tune on pre-trained BERT to perform conditional MLM task.”
AutoQA: From Databases to Q&A Semantic Parsers with Only Synthetic Training Data (2020, methodology): “Wu et al. (2019) used BERT (Devlin et al., 2019) in a similar way, and Hu et al. (2019b) improved upon it by jointly fine-tuning BERT and the classifier.”
Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Tasks (2020, methodology): “Other approaches generate synthetic data by using generative adversarial networks (Tanaka and Aranha, 2019), by using a language model to replace certain words (Wu et al., 2019) or to generate complete sentences (AnabyTavor et al., 2019).”
Multilingual Sentiment Analysis and Toxicity Detection for Text Messages in Russian (2021, methodology): “When augmenting textual data, it is important to preserve the meaning of the text, often by replacing words with syn-onyms using various dictionaries, for example, WordNet [22] or using pre-trained language models, such as BERT, GPT2, Word2Vec, Glove [23]–[25].”