Double Path Networks for Sequence to Sequence Learning

By Kaitao Song, Xu Tan, Di He, Jianfeng Lu, Tao Qin, Tie-Yan Liu. 2018

15 citations recorded by the source corpus.

Encoder-decoder based Sequence to Sequence learning (S2S) has made remarkable progress in recent years. Different network architectures have been used in the encoder/decoder. Among them, Convolutional Neural Networks (CNN) and Self Attention Networks (SAN) are the prominent ones. The two architectures achieve similar performances but use very different ways to encode and decode context: CNN use convolutional layers to focus on the local connectivity of the sequence, while SAN uses self-attention layers to focus on global semantics. In this work we propose Double Path Networks for Sequence to Sequence learning (DPN-S2S), which leverage the advantages of both models by using double path information fusion. During the encoding step, we develop a double path architecture to maintain the information coming from different paths with convolutional layers and self-attention layers separately. To effectively use the encoded context, we develop a gated attention fusion module and use it to automatically pick up the information needed during the decoding step, which is also a double path network. By deeply integrating the two paths, both types of information are combined and well exploited. Experiments show that our proposed method can significantly improve the performance of sequence to sequence learning over state-of-the-art systems.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tie-Yan Liu
2. Tao Qin
3. Myle Ott
4. Xu Tan
5. Michael Auli
6. Luke S. Zettlemoyer
7. Alexei Baevski
8. Angela Fan

In Computer Vision and Pattern Recognition

1. Michael Auli
2. Alexei Baevski
3. Tie-Yan Liu
4. Myle Ott
5. Tao Qin
6. Luc Van Gool
7. Angela Fan
8. Dacheng Tao

In Natural Language Processing

1. Michael Auli
2. Alexei Baevski
3. Myle Ott
4. Tie-Yan Liu
5. Tao Qin
6. Angela Fan
7. Luke S. Zettlemoyer
8. Michael Lewis

Citations in the corpus

• MASS: Masked Sequence to Sequence Pre-training for Language Generation (2019, background): “…Gehring et al., 2017; Vaswani et al., 2017; Tan et al., 2019; Artetxe et al., 2017; Lample et al., 2017; 2018; He et al., 2018; Hassan et al., 2018; Song et al., 2018; Shen et al., 2018), text summarization (Ayana et al., 2016; Suzuki & Nagata, 2017; Gehring et al., 2017), question answering (Yuan…”
• fairseq: A Fast, Extensible Toolkit for Sequence Modeling (2019, methodology): “FAIRSEQ has been used in many applications, such as machine translation (Gehring et al., 2017; Edunov et al., 2018b,a; Chen et al., 2018; Ott et al., 2018a; Song et al., 2018; Wu et al., 2019), language modeling (Dauphin et al.”
• Multilingual Neural Machine Translation with Knowledge Distillation (2019, background): “…to further improve the translation of a single language pair is a hot research topic (Vaswani et al., 2017; Gehring et al., 2017; Wu et al., 2018; Song et al., 2018; Gong et al., 2018), a new trend is multilingual neural machine translation (Dong et al., 2015; Luong et al., 2015a; Firat et al.,…”
• Sentence-wise Smooth Regularization for Sequence to Sequence Learning (2018, background): “…(NLP) tasks such as machine translation (Bahdanau, Cho, and Bengio 2014; Cho et al. 2014; Wu et al. 2016; Vaswani et al. 2017; Gehring et al. 2017; He et al. 2018; Song et al. 2018; Shen et al. 2018), text summarization (Rush et al. 2015; Shen et al. 2016) and dialog systems (Serban et al. 2016).”