Modeling Coverage for Neural Machine Translation

By Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, Hang Li. 2016

778 citations recorded by the source corpus.

Attention mechanism has enhanced state-of-the-art Neural Machine Translation (NMT) by jointly learning to align and translate. It tends to ignore past alignment information, however, which often leads to over-translation and under-translation. To address this problem, we propose coverage-based NMT in this paper. We maintain a coverage vector to keep track of the attention history. The coverage vector is fed to the attention model to help adjust future attention, which lets NMT system to consider more about untranslated source words. Experiments show that the proposed approach significantly improves both translation quality and alignment quality over standard attention-based NMT.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Massive Exploration of Neural Machine Translation Architectures (2017, background): “Recently, extensions such as coverage penalties (Tu et al., 2016) and length normalizations (Wu et al., 2016) have been shown to improve decoding results.”
Get To The Point: Summarization with Pointer-Generator Networks (2017, methodology): “Originating from Statistical Machine Translation (Koehn, 2009), coverage was adapted for NMT by Tu et al. (2016) and Mi et al. (2016), who both use a GRU to update the coverage vector each step.”
SPoC: Search-based Pseudocode to Code (2019, methodology): “To generate candidate code lines, we use a standard seq2seq translation model with an LSTM encoder and decoder [19], attention-based copying mechanism [24, 37], and coverage vector [34].”
Survey of Hallucination in Natural Language Generation (2022, background): “[175] show that hallucinations are visible in attention matrices.”
Semi-Supervised QA with Generative Domain-Adaptive Nets (2017, background): “In the context of machine translation, given a language pair, various recent work studied jointly training models to learn the mappings in both directions (Tu et al., 2016; Xia et al., 2016).”
Bottom-Up Abstractive Summarization (2018, methodology): “See et al. (2017) reﬁne this pointer-generator approach and use an additional coverage mechanism (Tu et al., 2016) that makes a model aware of its attention history to prevent repeated attention.”
Achieving Human Parity on Automatic Chinese to English News Translation (2018, methodology): “This indicates that there is still room to improve machine translation quality via various approaches, such as modeling Missing Words [36, 15], integration of high quality data for named-entity translation, as well as domain and topic adaptation for the issues of incorrect words and ungrammaticality”
Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategies (2018, methodology): “Pointer-Generator The pointer-generator model (See et al., 2017) uses abstractive token generation and extractive token copying using a pointer mechanism (Vinyals et al., 2015; G¨ulc¸ehre et al., 2016), keeping track of extractions using coverage (Tu et al., 2016).”
Towards Better Decoding and Language Model Integration in Sequence to Sequence Models (2016, methodology): “Past attentions vectors were used as auxiliary inputs in the emitting RNN either directly [34], or as cumulative coverage information [35].”
Six Challenges for Neural Machine Translation (2017, methodology): “For instance, prior work used the alignments provided by the attention model to interpolate word translation decisions with traditional probabilistic dic-r tionaries (Arthur et al., 2016), for the introduction of coverage and fertility models (Tu et al., 2016), etc.”
Masked Language Model Scoring (2019, background): “One can view these as exacerbations of the rare word problem due to overconﬁdent logits (Nguyen and Chiang, 2018), and of over-translation (Tu et al., 2016).”
Language to Logical Form with Neural Attention (2016, background): “This is a common problem for encoder-decoder models and can be addressed by explicitly modelling the decoding coverage of the source words (Tu et al., 2016; Cohn et al., 2016).”
Improved training of end-to-end attention models for speech recognition (2018, methodology): “Our model closely follows the machine translation model presented by Bahar et al. [51] and Bahdanau et al. [20] and we use a variant of attention weight / fertility feedback [52], which is inverse in our case, to use a multiplication instead of a division, for better numerical stability.”
Paragraph-level Neural Question Generation with Maxout Pointer and Gated Self-attention Networks (2018, background): “This issue exacerbates the repetition problem which has already been commonly observed in sequence to sequence models (Tu et al., 2016; See et al., 2017).”
Cutting-off Redundant Repeating Generations for Neural Abstractive Summarization (2016, methodology): “Note here that our method never requires ˜ r j [ m ] ≤ 0 (or ˜ r j [ m ] = 0 ) for all m at the last decoding time step j , as is generally required in the coverage Tu et al. (2016); Mi et al. (2016); Wu et al. (2016).”
Visualizing and Understanding Neural Machine Translation (2017, methodology): “In the future, we plan to apply our approach to more NMT approaches (Sutskever et al., 2014; Shen et al., 2016; Tu et al., 2016; Wu et al., 2016) on more language pairs to further verify its effectiveness.”
Search Engine Guided Non-Parametric Neural Machine Translation (2017, positive): “Motivated by the coverage penalty from (Tu et al., 2016), we propose to augment the bilinear matching function (in Eq.”