Lattice-Based Recurrent Neural Network Encoders for Neural Machine Translation

By Jinsong Su, Zhixing Tan, Deyi Xiong, Rongrong Ji, Xiaodong Shi, Yang Liu. 2016

92 citations recorded by the source corpus.

Neural machine translation (NMT) heavily relies on word-level modelling to learn semantic representations of input sentences.However, for languages without natural word delimiters (e.g., Chinese) where input sentences have to be tokenized first,conventional NMT is confronted with two issues:1) it is difficult to find an optimal tokenization granularity for source sentence modelling, and2) errors in 1-best tokenizations may propagate to the encoder of NMT.To handle these issues, we propose word-lattice based Recurrent Neural Network (RNN) encoders for NMT,which generalize the standard RNN to word lattice topology.The proposed encoders take as input a word lattice that compactly encodes multiple tokenizations, and learn to generate new hidden states from arbitrarily many inputs and hidden states in preceding time steps.As such, the word-lattice based encoders not only alleviate the negative impact of tokenization errors but also are more expressive and flexible to embed input sentences.Experiment results on Chinese-English translation demonstrate the superiorities of the proposed encoders over the conventional encoder.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Felix Stahlberg
2. Graham Neubig
3. Yang Liu
4. Jie Yang
5. Mohit Bansal
6. Kyunghyun Cho
7. Yoav Goldberg
8. Dong Yu

In Natural Language Processing

1. Felix Stahlberg
2. Graham Neubig
3. Jie Yang
4. A. Waibel
5. Jan Niehues
6. Yang Liu
7. Dong Yu
8. Rui Wang

Citations in the corpus

• Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates (2018, background): “Lattice-to-sequence models (Su et al., 2017; Sperber et al., 2017) are natural extension of sequence-to-sequence models, which represent inputs uncertainty through lattices.”
• Structural Information Preserving for Graph-to-Text Generation (2020, background): “…al., 2018; Cao and Clark, 2019; Damonte and Cohen, 2019; Ha-jdik et al., 2019; Koncel-Kedziorski et al., 2019; Hong et al., 2019; Song et al., 2019; Su et al., 2017) mainly studied how to effectively represent input graphs, and all these models are trained only with the standard language modeling…”
• Neural Lattice Language Models (2018, methodology): “Su et al. (2016) utilized a variation of the Gated Recurrent Unit that operated over lattices, and preprocessed lattices over Chinese characters that allowed it to effectively encode multiple segmentations.”
• Multi-Domain Neural Machine Translation with Word-Level Domain Context Discrimination (2018, methodology): “Therefore, we plan to apply our method to the NMT with complex architectures, for example, lattice-to-sequence NMT (Su et al., 2017), hierarchy-to-sequence NMT (Su et al., 2018), NMT with context-aware encoder (Zhang et al., 2017) and Transformer (Vaswani et al., 2017) and so on.”
• Chunk-based Decoder for Neural Machine Translation (2017, unknown): “tion into NMT (Eriguchi et al., 2016b; Su et al., 2017).”
• Lattice-Based Transformer Encoder for Neural Machine Translation (2019, methodology): “All the edges in the lattice G are the actual input tokens for NMT.”
• Neural Lattice-to-Sequence Models for Uncertain Inputs (2017, methodology): “This demonstrates a clear advantage of our proposed method over that of Su et al. (2017).”
• Neural Machine Translation: A Review (2019, background): “Lattices can represent uncertainty of upstream components such as speech recognizers [584] or tokenizers [585, 586].”
• Chinese NER Using Lattice LSTM (2018, methodology): “, 2017) and multi-granularity segmentation outputs (Su et al., 2017) for NMT encoders.”