Data Noising as Smoothing in Neural Network Language Models

By Ziang Xie, Sida I. Wang, Jiwei Li, Daniel Lévy, Allen Nie, Dan Jurafsky, Andrew Y. Ng. 2017

260 citations recorded by the source corpus.

Data noising is an effective technique for regularizing neural network models. While noising is widely adopted in application domains such as vision and speech, commonly used noising primitives have not been developed for discrete sequence-level settings such as language modeling. In this paper, we derive a connection between input noising in neural network language models and smoothing in $n$-gram models. Using this connection, we draw upon ideas from smoothing to develop effective noising schemes. We demonstrate performance gains when applying the proposed schemes to language modeling and machine translation. Finally, we provide empirical analysis validating the relationship between noising and smoothing.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates (2018, background): “(Xie et al., 2017) also proposes a robust RNN language model that interpolates random unigram language model.”
BPE-Dropout: Simple and Effective Subword Regularization (2019, methodology): “, based on token frequency or a language model) (Fadaee et al., 2017; Xie et al., 2017; Kobayashi, 2018).”
AutoML: A Survey of the State-of-the-Art (2019, neutral): “[40] have proposed a non-domain-specific”
EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks (2019, methodology): “Other works have used predictive language models for synonym replacement (Kobayashi, 2018) and data noising as smoothing (Xie et al., 2017).”
Conditional BERT Contextual Augmentation (2018, methodology): “In some speciﬁc areas (Jia and Liang, 2017; Xie et al., 2017; Ebrahimi et al., 2017). word replacement augmentation was applied.”
Noising and Denoising Natural Language: Diverse Backtranslation for Grammar Correction (2018, background): “Some recent work has proposed noising— in the form of dropping or replacing individual tokens—as a regularizer when training sequence models, where it has been demonstrated to have a smoothing effect on the softmax output distribution (Bowman et al., 2015; Xie et al., 2017; Dai and Le, 2015; Kumar e”
Toward Robust Neural Machine Translation for Noisy Input Sequences (2017, methodology): “Inducing noise in the training inputs can be seen as a form of data augmentation, which has been used in several applications such as acoustic modeling [16], computer vision [17], language modeling [18], and statistical machine translation where data can be augmented by paraphrases [19].”
A Survey on Data Augmentation for Text Classification (2021, unknown): “Data augmentation methods in computer vision are, among other, geometric transformations [7, 24], neural style transfers [31–33], interpolation of images [34], random partial deletions [35], and generative adversarial network ( GAN ) data generation [21].”
Robust Multilingual Part-of-Speech Tagging via Adversarial Training (2017, background): “Xie et al. (2017) discuss various data noising techniques for language modeling.”
Data Augmentation Approaches in Natural Language Processing: A Survey (2021, methodology): “[23] borrow from the idea of “word-dropout” and improve generalization by reducing the information in the sentence.”
Learning Data Manipulation for Augmentation and Weighting (2019, background): “In addition to perturbing the input text as in classification tasks, text generation problems expose opportunities to adding noise also in the output text, such as [35, 51].”
Adversarial Training for Relation Extraction (2017, background): “Xie et al. (2017) discusses various data noising techniques for language models.”
Semantic Equivalent Adversarial Data Augmentation for Visual Question Answering (2020, background): “Other works generate paraphrases [45,28] and add noise to smooth text data [42].”
Progressive Generation of Long Text with Pretrained Language Models (2021, methodology): “To alleviate the issue and increase the robustness of each intermediate LM, we draw on the rich literature of addressing token-level exposure bias (Xie et al., 2017; Tan et al., 2019).”
Data Boost: Text Data Augmentation through Reinforcement Learning Guided Conditional Generation (2020, methodology): “Naive methods imitate pixel manipulation in CV, augmenting sentences by adding spelling errors (Xie et al., 2017), or randomly deleting and swapping tokens (Wei and Zou, 2019).”
Anonymized BERT: An Augmentation Approach to the Gendered Pronoun Resolution Challenge (2019, background): “Although augmentation has been shown to be very effective in deep learning (Xie et al., 2019), most NLP augmentation methods are on document or sentence level, such as synonym replacement (Zhang et al., 2015), data noising (Xie et al., 2017) and back-translation (Yu et al., 2018).”
SSMBA: Self-Supervised Manifold Based Data Augmentation for Improving Out-of-Domain Robustness (2020, methodology): “, 2017) and word-level methods (Zhang et al., 2015; Xie et al., 2017; Wei and Zou, 2019) are based on human-designed heuristics.”
Data Manipulation: Towards Effective Instance Learning for Neural Dialogue Generation via Learning to Augment and Reweight (2020, background): “Xie et al. (2017); Andreas (2020) design manuallyspecified strategies for data augmentation.”
SeqVAT: Virtual Adversarial Training for Semi-Supervised Sequence Labeling (2020, background): “Prior to AT, several papers investigated various ways of perturbations (Xie et al., 2017).”