Robust Training under Linguistic Adversity

By Timothy Baldwin, Trevor Cohn, Yitong Li. 2017

61 citations recorded by the source corpus.

Deep neural networks have achieved remarkable results across many language processing tasks, however they have been shown to be susceptible to overfitting and highly sensitive to noise, including adversarial attacks. In this work, we propose a linguistically-motivated approach for training robust models based on exposing the model to corrupted text examples at training time. We consider several flavours of linguistically plausible corruption, include lexical semantic and syntactic methods. Empirically, we evaluate our method with a convolutional neural model across a range of sentiment analysis datasets. Compared with a baseline and the dropout method, our method achieves better overall performance.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

A Survey of Data Augmentation Approaches for NLP (2021, background): “but usually offer incremental performance improvements (Li et al., 2017; Wei and Zou, 2019; Wei et al., 2021b).”
Few-Shot Text Classification with Triplet Networks, Data Augmentation, and Curriculum Learning (2021, background): “…vision where augmented data undoubtedly resembles original data, in text, data augmentation techniques might introduce linguistic adversity and therefore can be seen as a form of noising (Li et al., 2017; Wang et al., 2018), where noised data is harder to learn from than unmodified original data.”
Improving the Interpretability of Neural Sentiment Classifiers via Data Augmentation (2019, methodology): “category of augmented data, are mainly used to enhance model robustness (Li et al., 2017; Sun et al., 2018).”
A Survey on Data Augmentation for Text Classification (2021, unknown): “[41] Substitutable words are nouns, verbs, adjectives, or adverbs that are not part of a named entity.”
Clinical Text Data in Machine Learning: Systematic Review (2020, methodology): “Similar techniques are now increasingly used to augment text data in a quest to improve generalization performance of the corresponding machine learning models [140-143].”
Text Augmentation in a Multi-Task View (2021, background): “(Wang and Yang, 2015; Zhang et al., 2015; Li et al., 2017); Pervasive Dropout randomly removes words from the input sequence (Sennrich et al., 2016a); and SwitchOut (for machine translation) replaces some words in both source and target sentences with other words from their corresponding…”
How Robust Are Character-Based Word Embeddings in Tagging and MT Against Wrod Scramlbing or Randdm Nouse? (2017, background): “, 2013; Plank, 2016) and in particular data noising (Yitong et al., 2017), do not have a long and extensive history in NLP.”
GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation (2021, methodology): “The technique has been very effective on image data, but it has been harder to establish a standard approach for texts due to the inherent sparse nature of linguistic distributions, which attributes to the challenges of identifying adversarial text examples (Li et al., 2017).”
Achieving Verified Robustness to Symbol Substitutions via Interval Bound Propagation (2019, background): “Common attempts to mitigate the issue are adversarial training (Ebrahimi et al., 2018) and data augmentation (Belinkov and Bisk, 2018; Li et al., 2017), which lead to improved accuracy on adversarial examples.”
Adversarial Training for Relation Extraction (2017, methodology): “Connection to other approaches: Li et al. (2017); Xie et al. (2017) proposes linguistic adversaries techniques to enhance the robustness of the model by randomly changing the word tokens in a sentence.”
Improving short text classification through global augmentation methods (2019, background): “the nal model. Augmentation of text data to be able to create robust models has had dierent factors of success. Some approaches require more direct information about the language at hand than others [19], while others are more agnostic given a learned language model that can be used [16]. Over the l”