A Comparative Study on Regularization Strategies for Embedding-based Neural Networks

By Hao Peng, Lili Mou, Ge Li, Yunchuan Chen, Yangyang Lu, Zhi Jin. 2015

29 citations recorded by the source corpus.

This paper aims to compare different regularization strategies to address a common phenomenon, severe overfitting, in embedding-based neural networks for NLP. We chose two widely studied neural models and tasks as our testbed. We tried several frequently applied or newly proposed regularization strategies, including penalizing weights (embeddings excluded), penalizing embeddings, re-embedding words, and dropout. We also emphasized on incremental hyperparameter tuning, and combining different regularizations. The results provide a picture on tuning hyperparameters for neural NLP models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Ge Li
2. Zhi Jin
3. Lili Mou
4. Rui Yan
5. Lu Zhang
6. Yan Xu
7. Ye Zhang
8. Yong Li

In Computer Vision and Pattern Recognition

1. Ye Zhang
2. Yong Li
3. Chen Gao
4. Byron C. Wallace
5. Ge Li
6. Bei Chen
7. Rui Yan
8. Zhi Jin

Citations in the corpus

• A Sensitivity Analysis of (and Practitioners’ Guide to) Convolutional Neural Networks for Sentence Classification (2015, background): “, 2014), and also in contrast to (Peng et al., 2015), which uses recursive neural network (RNN) for sentiment analysis and applies regularization in all layers.”
• How Transferable are Neural Networks in NLP Applications? (2016, background): “Existing evidence shows that frozen parameters would generally hurt the performance (Peng et al., 2015), but this setting provides a more direct understanding on how transferable the features are (because the factor of target domain optimization is ruled out).”
• Natural Language Inference by Tree-Based Convolution and Heuristic Matching (2015, background): “We see that a large dropout rate ( ≥ 0.3) hurts the performance (and also makes training slow) for such a large dataset as opposed to small datasets in other tasks (Peng et al., 2015).”
• λOpt: Learn to Regularize Recommender Models in Finer Levels (2019, background): “In natural language processing, training large embeddings usually requires suitable regularization [25].”