Shortcut Sequence Tagging

By Huijia Wu, Jiajun Zhang, Chengqing Zong. 2017

Deep stacked RNNs are usually hard to train. Recent studies have shown that shortcut connections across different RNN layers bring substantially faster convergence. However, shortcuts increase the computational complexity of the recurrent computations. To reduce the complexity, we propose the shortcut block, which is a refinement of the shortcut LSTM blocks. Our approach is to replace the self-connected parts (\(c_t^l\)) with shortcuts (\(h_t^{l-2}\)) in the internal states. We present extensive empirical experiments showing that this design performs better than the original shortcuts. We evaluate our method on CCG supertagging task, obtaining a 8% relatively improvement over current state-of-the-art results.

Citations in the corpus

• Semi-Supervised Sequence Modeling with Cross-View Training (2018, neutral): “(Chiu and Nichols, 2016) 91.62 ± 0.33 86.28 ± 0.26 ID-CNN-CRF (Strubell et al., 2017) 90.65 ± 0.15 86.84 ± 0.19 Tri-Trained LSTM (Lewis et al., 2016) 94.7 Shortcut LSTM (Wu et al., 2017) 95.08 97.53 JMT* (Hashimoto et al., 2017) 95.77 97.55 94.67 92.90 LM-LSTM-CNN-CRF (Liu et al., 2017) 95.96 ± 0.08”