Lattice-Based Transformer Encoder for Neural Machine Translation

By Fengshun Xiao, Jiangtong Li, Zhao Hai, Rui Wang, Kehai Chen. 2019

45 citations recorded by the source corpus.

Neural machine translation (NMT) takes deterministic sequences for source representations. However, either word-level or subword-level segmentations have multiple choices to split a source sequence with different word segmentors or different subword vocabulary sizes. We hypothesize that the diversity in segmentations may affect the NMT performance. To integrate different segmentations with the state-of-the-art NMT model, Transformer, we propose lattice-based encoders to explore effective word or subword representation in an automatic way during training. We propose two methods: 1) lattice positional encoding and 2) lattice-aware self-attention. These two methods can be used together and show complementary to each other to further improve translation performance. Experiment results show superiorities of lattice-based encoders in word-level and subword-level representations over conventional Transformer encoder.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhuosheng Zhang
2. Xipeng Qiu
3. Xuanjing Huang
4. Yuwei Wu
5. Xi Zhou
6. Hang Yan
7. Xiaoping Zhou
8. Junru Zhou

In Computer Vision and Pattern Recognition

1. Hang Yan
2. Xipeng Qiu
3. Xuanjing Huang
4. Zhuosheng Zhang
5. Zhao Hai
6. Xiaonan Li
7. Xiaoping Zhou

In Natural Language Processing

1. Zhuosheng Zhang
2. Xipeng Qiu
3. Xuanjing Huang
4. Yuwei Wu
5. Xi Zhou
6. Xiaoping Zhou
7. Zhao Hai
8. Junru Zhou

Citations in the corpus

• Dual Co-Matching Network for Multi-choice Reading Comprehension (2019, methodology): “…which can be well pre-trained language models such as BERT (Devlin et al. 2019) and XLNet (Yang et al. 2019), as using transformer as the contextualized encoder has shown to be very powerful in language representation (Zhang et al. 2019a; Zhou and Zhao ; Luo, Xiao, and Zhao 2019; Xiao et al. 2019).”
• Hierarchical Contextualized Representation for Named Entity Recognition (2019, background): “Neural Named Entity Recognition Recently, with the development of deep neural network in a wide range of NLP tasks (He et al. 2018; He, Li, and Zhao 2019; Zhou and Zhao 2019; Xiao et al. 2019; Zhang et al. 2020a; 2020b), neural network based models build reliable NER systems without hand-crafted fea”
• FLAT: Chinese NER Using Flat-Lattice Transformer (2020, background): “In Chinese-source translation, Xiao et al. (2019) take the absolute position of nodes’ first characters and the relation between each pair of nodes as the structure information.”
• Relative Positional Encoding for Transformers with Linear Complexity (2021, unknown): “Tree-based PE was proposed both for APE (Shiv & Quirk, 2019; Xiao et al., 2019; Ma et al., 2019) and RPE (Omote et al.”
• Head-Driven Phrase Structure Grammar Parsing on Penn Treebank (2019, background): “…NLP tasks (Li et al., 2018b; Zhang et al., 2018a; Li et al., 2018c; Zhang et al., 2018c,b; Zhang and Zhao, 2018; Cai et al., 2018; He et al., 2018; Xiao et al., 2019; Chen et al., 2018; Wang et al., 2018; Wang et al., 2018a, 2017b,a), constituent and dependency parsing have been well developed…”