Structure Regularization for Structured Prediction

By Xu Sun. 2014

63 citations recorded by the source corpus.

While there are many studies on weight regularization, the study on structure regularization is rare. Many existing systems on structured prediction focus on increasing the level of structural dependencies within the model. However, this trend could have been misdirected, because our study suggests that complex structures are actually harmful to generalization ability in structured prediction. To control structure-based overfitting, we propose a structure regularization framework via structure decomposition, which decomposes training samples into mini-samples with simpler structures, deriving a model with better generalization power. We show both theoretically and empirically that structure regularization can effectively control overfitting risk and lead to better accuracy. As a by-product, the proposed method can also substantially accelerate the training speed. The method and the theoretical results can apply to general graphical models with arbitrary structures. Experiments on well-known tasks demonstrate that our method can easily beat the benchmark systems on those highly-competitive tasks, achieving record-breaking accuracies yet with substantially faster training speed.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jiawei Han
2. Xiang Ren
3. Kai Yu
4. Wei Xu
5. Zhiheng Huang
6. Houfeng Wang
7. Liyuan Liu
8. Xuancheng Ren

Citations in the corpus

• Bidirectional LSTM-CRF Models for Sequence Tagging (2015, background): “…learning (Shen et al., 2007) 97.33 No Semi-supervised condensed nearest neighbor 97.50 Yes (Soegaard, 2011) CRFs with structure regularization (Sun, 2014) 97.36 No Conv network tagger (Collobert et al., 2011) 96.37 No Conv network tagger (senna) (Collobert et al., 2011) 97 (Kudo and…”
• Empower Sequence Labeling with Task-Aware Neural Language Model (2017, background): “We also list some top reported performance on each dataset (Collobert et al. 2011; Luo et al. 2015; Chiu and Nichols 2016; Yang, Salakhutdinov, and Cohen 2017; Peters et al. 2017; Manning 2011; Søgaard and Goldberg 2016; Sun 2014).”
• meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting (2017, background): “The POS tagging task is a well-known benchmark task, with the accuracy reports from 97.2% to 97.4% (Toutanova et al., 2003; Sun, 2014; Shen et al., 2007; Tsuruoka et al., 2011; Collobert et al., 2011; Huang et al., 2015).”