Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relations

By Sosuke Kobayashi. 2018

682 citations recorded by the source corpus.

We propose a novel data augmentation for labeled sentences called contextual augmentation. We assume an invariance that sentences are natural even if the words in the sentences are replaced with other words with paradigmatic relations. We stochastically replace words with other words that are predicted by a bi-directional language model at the word positions. Words predicted according to a context are numerous but appropriate for the augmentation of the original words. Furthermore, we retrofit a language model with a label-conditional architecture, which allows the model to augment sentences without breaking the label-compatibility. Through the experiments for six various different text classification tasks, we demonstrate that the proposed method improves classifiers based on the convolutional or recurrent neural networks.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

A Survey of Data Augmentation Approaches for NLP (2021, methodology): “, 2019) × 3 const Input 3 CONTEXTUALAUG (Kobayashi, 2018) × 3 Input 3 LAMBADA (Anaby-Tavor et al.”
Do Not Have Enough Data? Deep Learning to the Rescue! (2019, neutral): “…apply transformations on existing training examples, such as cropping, padding, ﬂip-ping, and shifting along time and space dimensions, as these transformations are usually class preserving (Krizhevsky, Sutskever, and Hinton 2012; Cui, Goel, and Kingsbury 2015; Ko et al. 2015; Szegedy et al. 2015).”
BPE-Dropout: Simple and Effective Subword Regularization (2019, background): “, based on token frequency or a language model) (Fadaee et al., 2017; Xie et al., 2017; Kobayashi, 2018).”
Data Augmentation using Pre-trained Transformer Models (2020, methodology): “Further, Kobayashi (2018) utilized language models (LM) to augment training data.”
EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks (2019, background): “, 2011) for temporal analysis (7)(Kobayashi, 2018) for text classification (8)(Wang and Yang, 2015) for tweet classification (9)EDA does use a synonym dictionary, WordNet, but the cost of downloading it is far less than training a model on an external dataset, so we don’t count it as an “external da”
Conditional BERT Contextual Augmentation (2018, methodology): “For comparison, following Kobayashi(Kobayashi, 2018), we adopt two typical classiﬁer architectures: CNN or LSTM-RNN.”
G-DAug: Generative Data Augmentation for Commonsense Reasoning (2020, background): “, 2015; Wei and Zou, 2019) and word embeddings (Wang and Yang, 2015; Fadaee et al., 2017; Kobayashi, 2018; Wu et al., 2019), and very recently, generative models for synthesizing”
SeqMix: Augmenting Active Sequence Labeling via Sequence Mixup (2020, methodology): “Very recently, (Anaby-Tavor et al., 2020; Kobayashi, 2018) harness the power of pre-trained language models and augmenting the text data based on contextual patterns.”
Natural Perturbation for Robust Question Answering (2020, background): “There is a handful of work that studies semi-automatic contextual augmentation (Kobayashi, 2018; Cheng et al., 2018), often with the goal of creating better systems.”
Improving the Interpretability of Neural Sentiment Classifiers via Data Augmentation (2019, methodology): “Prior work on data augmentation for text classification (Kobayashi, 2018; Wei and Zou, 2019) mainly focuses more on improving prediction performance.”
Quantifying the Evaluation of Heuristic Methods for Textual Data Augmentation (2020, methodology): “…different approaches for text data augmentation; for example, (Zhang et al., 2015; Wei and Zou, 2019) used thesaurus-based and (Wang and Yang, 2015; Kobayashi, 2018; Jiao et al., 2019) used embedding-based lexical substitution approach, (Wei and Zou, 2019; Xie et al., 2019) used random noise…”
A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios (2020, background): “that takes context into consideration (Kobayashi, 2018; Fadaee et al., 2017).”
Improving short text classification through global augmentation methods (2019, methodology): “ierent factors of success. Some approaches require more direct information about the language at hand than others [19], while others are more agnostic given a learned language model that can be used [16]. Over the last few years, the development of distributed word representations (word embeddings) ”
A Diverse Data Augmentation Strategy for Low-Resource Neural Machine Translation (2020, background): “randomly replace word tokens sampled from the output distribution of one language model [23,26].”
The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A" (2023, methodology): “Similar approaches are used in the model augmentations literature (Sennrich et al., 2016; Cai et al., 2020; Kobayashi, 2018; Eldan & Li, 2023).”
A Survey of Deep Learning for Scientific Discovery (2020, background): “This latter strategy is usually guided by word embeddings [239] or contextualized word embeddings [110].”