Feature Noising for Log-Linear Structured Prediction

By Sida I. Wang, Mengqiu Wang, Stefan Wager, Percy Liang, Christopher D. Manning. 2013

30 citations recorded by the source corpus.

NLP models have many and sparse features, and regularization is key for balancing model overfitting versus underfitting. A recently repopularized form of regularization is to generate fake training data by repeatedly adding noise to real data. We reinterpret this noising as an explicit regularizer, and approximate it with a second-order formula that can be used during training without actually generating fake data. We show how to apply this method to structured prediction using multinomial logistic regression and linear-chain CRFs. We tackle the key challenge of developing a dynamic program to compute the gradient of the regularizer efficiently. The regularizer is a sum over inputs, so we can estimate it more accurately via a semi-supervised or transductive extension. Applied to text classification and NER, our method provides a >1% absolute performance gain over use of standardL2 regularization.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Dropout Training as Adaptive Regularization (2013, methodology): “In follow-up research [13], we extend the results from this paper to more complicated structured prediction, such as multi-class logistic regression and linear chain conditional random ﬁelds.”
To Drop or Not to Drop: Robustness, Consistency and Differential Privacy Properties of Dropout (2015, background): “A popular heuristic to avoid such local minima is dropout which perturbs the objective function randomly by dropping out several nodes of the DBN. Recently, there has been some work to understand this heuristic in certain limited convex settings Baldi and Sadowski [2014], Wager et al. [2013].”
Data Noising as Smoothing in Neural Network Language Models (2017, background): “Feature noising has been demonstrated to be effective for structured prediction tasks, and has been interpreted as an explicit regularizer (Wang et al., 2013).”
Dropout Training of Matrix Factorization and Autoencoder for Link Prediction in Sparse Graphs (2015, methodology): “Dropout has also been applied to training generalized linear models [21], log linear models with structured output [28], and distance metric learning [25].”
Altitude Training: Strong Bounds for Single-Layer Dropout (2014, background): “…et al., 2012, Wan et al., 2013], but it has also been found to improve the performance of logistic regression and other single-layer models for natural language tasks such as document classification and named entity recognition [Wager et al., 2013, Wang and Manning, 2013, Wang et al., 2013].”
Data Augmentation via Levy Processes (2016, background): “…single-layer models, where it is also known as “blankout noise” (Globerson and Roweis, 2006; van der Maaten et al., 2013) and has been successful in natural language tasks such as document classification and named entity recognition (Wager et al., 2013; Wang and Manning, 2013; Wang et al., 2013).”