Document-Level Neural Machine Translation with Hierarchical Attention Networks

By Lesly Miculicich, Dhananjay Ram, Nikolaos Pappas, James Henderson. 2018

292 citations recorded by the source corpus.

Neural Machine Translation (NMT) can be improved by including document-level contextual information. For this purpose, we propose a hierarchical attention model to capture the context in a structured and dynamic manner. The model is integrated in the original NMT architecture as another level of abstraction, conditioning on the NMT model’s own previous hidden states. Experiments show that hierarchical attention significantly improves the BLEU score over a strong NMT baseline with the state-of-the-art in context-aware methods, and that both the encoder and decoder benefit from context in complementary ways.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Multilingual Denoising Pre-training for Neural Machine Translation (2020, methodology): “1), without using any task-specific techniques developed by previous work (Miculicich et al., 2018; Li et al., 2019), such as constrained contexts or restricted attention.”
ETC: Encoding Long and Structured Inputs in Transformers (2020, methodology): “Moreover, notice that this idea of processing the input hierarchically is not speciﬁc to Transformer models, and it has been applied to recurrent neural network models both at the level of sentences (Yang et al., 2016; Miculicich et al., 2018) and blocks (Shen et al., 2018).”
Blockwise Self-Attention for Long Document Understanding (2019, methodology): “, 2019b) and document-level machine translation (Miculicich et al., 2018), and also non-NLP tasks such as protein sequence modeling (Rives et al.”
Random Feature Attention (2021, background): “These results suggest that R FA can be particularly useful in sequence-to-sequence tasks with longer sequences, e.g., document-level machine translation (Miculicich et al., 2018).”
ETC: Encoding Long and Structured Data in Transformers (2020, methodology): “Notice that this idea of processing the input hierarchically is not specific to Transformer models, and it has been applied to recurrent neural network models both at the level of sentences (Yang et al., 2016; Miculicich et al., 2018) and blocks (Shen et al., 2018).”
Investigating Failures of Automatic Translation in the Case of Unambiguous Gender (2021, background): “It is well known that translation systems perform better when provided with more context (i.e., longer sentences; Tiedemann and Scherrer 2017; Miculicich et al. 2018), so in our simpler setting, we augment our sentences with statistically gender-indicative adjectives (such as pretty and handsome, th”
BP-Transformer: Modelling Long-Range Context via Binary Partitioning (2019, methodology): “(1) Hierarchical Transformers (Miculicich et al., 2018; Liu and Lapata, 2019) uses two Transformers in a hierarchical architecture: one Transformer models the sentence representation with word-level context, and another the document representation with the sentence-level context.”
Pyraformer: Low-Complexity Pyramidal Attention for Long-Range Time Series Modeling and Forecasting (2022, methodology): “HIBERT (Miculicich et al., 2018) first uses a Sent Encoder to extract the features of a sentence, and then forms the EOS tokens of sentences in the document as a new sequence and input it into the Doc Encoder.”
Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense (2023, background): “…machine translation [House, 2006, Jean et al., 2017, Wang et al., 2017, Tiedemann and Scherrer, 2017, Kuang et al., 2018, Agrawal et al., 2018, Miculicich et al., 2018, Zhang et al., 2018, Xiong et al., 2019, Jean et al., 2019, Voita et al., 2019a, Yin et al., 2021, Mansimov et al., 2021].”
Pretrained Language Models for Document-Level Neural Machine Translation (2019, background): “Another ﬁnding is that using smaller contexts achieves a signiﬁcantly better BLEU score (+1.0) than larger contexts, similar to Zhang et al. (2018); Miculicich et al. (2018).”
Incorporating BERT into Neural Machine Translation (2020, unknown): “(2) Another baseline is proposed by Miculicich et al. (2018), where multiple preceding sentences in a document are leveraged using a hierarchical attention network.”
Putting Machine Translation in Context with the Noisy Channel Model (2019, methodology): “…NMT typically adapts the MT system to take additional context as input, either a few previous sentences (Jean et al., 2017; Wang et al., 2017; Tu et al., 2018; Titov et al., 2018; Zhang et al., 2018; Werlen et al., 2018) or the full document (Haffari & Maruf, 2018; Maruf et al., 2019).”
Data augmentation using back-translation for context-aware neural machine translation (2019, background): “…Scherrer, 2017; Bawden et al., 2018) and multi-encoder models that design an additional context encoder to process the contexts (Jean et al., 2017; Wang et al., 2017; Baw-den et al., 2018; Voita et al., 2018; Maruf and Haf-fari, 2018; Miculicich et al., 2018; Tu et al., 2018; Maruf et al., 2019).”
A Survey of Transformers (2021, background): “[92] introduce dependencies on the previous sentences from both the source and target sides when translating a sentence.”
Better Document-Level Machine Translation with Bayes’ Rule (2019, methodology): “Existing work in the area of context-aware NMT typically adapts the MT system to take additional context as input, either a few previous sentences (Jean et al., 2017; Wang et al., 2017; Tu et al., 2018; Voita et al., 2018; Zhang et al., 2018; Werlen et al., 2018) or the full document (Haffari andMar”
Encouraging Lexical Translation Consistency for Document-Level Neural Machine Translation (2021, methodology): “We follow Miculicich et al. (2018) and Tan et al. (2019) to evaluate coreference and anaphora using the reference-based metric: accuracy of pronoun translation (Werlen and Popescu-Belis, 2017).”
The University of Helsinki Submissions to the WMT19 News Translation Task (2019, methodology): “• NMT-HAN: Sentence-level transformer model with a hierarchical attention network to capture the document-level context (Miculicich et al., 2018).”