Sentence-wise Smooth Regularization for Sequence to Sequence Learning

By Chengyue Gong, Xu Tan, Di He, Tao Qin. 2018

9 citations recorded by the source corpus.

Maximum-likelihood estimation (MLE) is widely used in sequence to sequence tasks for model training. It uniformly treats the generation/prediction of each target token as multiclass classification, and yields non-smooth prediction probabilities: in a target sequence, some tokens are predicted with small probabilities while other tokens are with large probabilities. According to our empirical study, we find that the non-smoothness of the probabilities results in low quality of generated sequences. In this paper, we propose a sentence-wise regularization method which aims to output smooth prediction probabilities for all the tokens in the target sequence. Our proposed method can automatically adjust the weights and gradients of each token in one sentence to ensure the predictions in a sequence uniformly well. Experiments on three neural machine translation tasks and one text summarization task show that our method outperforms conventional MLE loss on all these tasks and achieves promising BLEU scores on WMT14 English-German and WMT17 Chinese-English translation task.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tao Qin
2. Xu Tan
3. Tie-Yan Liu
4. Yi Ren
5. Sheng Zhao
6. Zhou Zhao
7. Qiang Liu
8. Dilin Wang

Citations in the corpus

• Improving Neural Language Modeling via Adversarial Training (2019, methodology): “We use the processed version provided by Mikolov et al. (2010) that is widely used for this dataset (e.g., Mer-ity et al., 2018a; Yang et al., 2018; Kanai et al., 2018; Gong et al., 2019).”
• Multilingual Neural Machine Translation with Knowledge Distillation (2019, background): “Neural Machine Translation (NMT) has witnessed rapid development in recent years (Bahdanau et al., 2015; Luong et al., 2015b; Wu et al., 2016; Gehring et al., 2017; Vaswani et al., 2017; Wu et al., 2018; Song et al., 2018; Shen et al., 2018; Guo et al., 2018; He et al., 2018; Gong et al., 2018), inc”
• Token-Level Ensemble Distillation for Grapheme-to-Phoneme Conversion (2019, methodology): “Besides, we adopt Transformer [9] instead of RNN or CNN as the basic encoder-decoder model structure, since it demonstrates advantages in a variety of sequence to sequence tasks, such as neural machine translation [9], text summarization [10], automatic speech recognition [11].”
• Non-Autoregressive Neural Machine Translation with Enhanced Decoder Input (2018, background): “By doing so, we provide less noisy and more deterministic training data which make the NAT model easy to learn (Kim and Rush 2016; Ott et al. 2018; Gong et al. 2019).”