Neural Machine Translation with Reconstruction

By Zhaopeng Tu, Yang Liu, Lifeng Shang, Xiaohua Liu, Hang Li. 2016

210 citations recorded by the source corpus.

Although end-to-end Neural Machine Translation (NMT) has achieved remarkable progress in the past two years, it suffers from a major drawback: translations generated by NMT systems often lack of adequacy. It has been widely observed that NMT tends to repeatedly translate some source words while mistakenly ignoring other words. To alleviate this problem, we propose a novel encoder-decoder-reconstructor framework for NMT. The reconstructor, incorporated into the NMT model, manages to reconstruct the input source sentence from the hidden layer of the output target sentence, to ensure that the information in the source side is transformed to the target side as much as possible. Experiments show that the proposed framework significantly improves the adequacy of NMT output and achieves superior translation result over state-of-the-art NMT and statistical MT systems.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Massive Exploration of Neural Machine Translation Architectures (2017, background): “It has also been observed (Tu et al., 2017) that very large beam sizes, even with length penalty, perform worse than smaller ones.”
Survey of Hallucination in Natural Language Generation (2022, background): ““Faithfulness” refers to the amount of source meaning that is faithfully expressed in the translation, and it is used interchangeably with the term “adequacy” [49, 186].”
Challenges in Data-to-Document Generation (2017, methodology): “A fully differentiable approach using the decoder hidden states has recently been successfully applied to neural machine translation by Tu et al. (2017).”
Lexically Constrained Decoding for Sequence Generation Using Grid Beam Search (2017, background): “Beam size in these models is also typically kept small, and recent work has shown that the performance of some architectures can actually degrade with larger beam size (Tu et al., 2016).”
Dual Transfer Learning for Neural Machine Translation with Marginal Distribution Regularization (2018, methodology): “Training Procedure Following (Tu et al. 2017; He et al. 2016a), to speed up training, for each task, we first trained NMT models on their own parallel corpora and then used them to initialize our algorithm.”
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization (2020, background): “Recently, more work has focused on semantic errors in model outputs, such as adequacy in machine translation (Tu et al., 2017), faithfulness in summarization (Cao et al., 2018), and consistency in dialogue (Li et al., 2019).”
The Neural Noisy Channel (2016, background): “Tu et al. (2016) propose augmenting a direct model’s decoding objective with a reverse translation model (similar to our channel model except it conditions on the direct model’s output RNN’s hidden states rather than the words); however, that work just reranks complete translation hypotheses rather…”
Towards Enhancing Faithfulness for Neural Machine Translation (2020, background): “Tu et al. (2017) proposed to introduce a reconstruction loss to make translation can reconstruct the input sentence.”
Simple Fusion: Return of the Language Model (2018, background): “For example, Cheng et al. (2016); Tu et al. (2017) proposed to add autoencoder terms to the training objective which capture how well a sentence can be reconstructed from its translated representation.”
Reconstruction Network for Video Captioning (2018, background): “In [37], the source sentences are repro-Figure duced from the target side hidden states, and the accuracy of reconstructed source provides a constraint for decoder to embed more information of source language into target language.”
Structural Information Preserving for Graph-to-Text Generation (2020, background): “Wiseman et al. (2017) extended the reconstruction loss of Tu et al. (2017) on table-to-text generation, where a table contains multiple records that ﬁt into several ﬁelds.”
Trainable Greedy Decoding for Neural Machine Translation (2017, background): “For instance, Tu et al. (2016a) showed that beam search with a very large beam, which is supposed to ﬁnd translations with better log-probabilities, suffers from pathological translations of very short length, resulting in low translation quality.”
Learning to Jointly Translate and Predict Dropped Pronouns with a Shared Reconstruction Mechanism (2018, methodology): “They employ two separate reconstructors (Tu et al., 2017) to respectively reconstruct encoder and decoder representations back to the DP-annotated source sentence.”
Source-side Prediction for Neural Headline Generation (2017, methodology): “The coverage model [Mi et al., 2016, Tu et al., 2016] enforces the decoder to attend to every part of the source sequence to translate all semantic information in the source.”
Neural Models of Text Normalization for Speech Applications (2019, background): “Tu et al. (2016) and Mi et al. (2016) argue that the root problem lies in the attention mechanism itself.”
Learning to Remember Translation History with a Continuous Cache (2017, background): “…which means our model is required to train all the parameters and the other models cannot use mini-batch sorting, the training speeds for the models listed in Table 5 are 728.8, 159.8, 572.4, and 627.3, respectively. domain (Tu et al., 2016; Li et al., 2017; Zhou et al., 2017; Wang et al., 2017c).”
Sequence to Sequence Mixture Model for Diverse Machine Translation (2018, result): “Similar observations have been made in the previous work (Tu et al., 2017).”