Attention Strategies for Multi-Source Sequence-to-Sequence Learning

By Jindřich Libovický, Jindřich Helcl. 2017

192 citations recorded by the source corpus.

Modeling attention in neural multi-source sequence-to-sequence learning remains a relatively unexplored area, despite its usefulness in tasks that incorporate multiple source languages or modalities. We propose two novel approaches to combine the outputs of attention mechanisms over each source sequence, flat and hierarchical. We compare the proposed methods with existing techniques and present results of systematic evaluation of those methods on the WMT16 Multimodal Translation and Automatic Post-editing tasks. We show that the proposed methods achieve competitive results on both tasks.

Subfields: Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Model Architectures

Citations in the corpus

VaTeX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research (2019, background): “For instance, [27, 23] consider the object features of the images, and [10, 34] import convolutional image features into machine translation.”
Probing the Need for Visual Context in Multimodal Machine Translation (2019, background): “, 2016) and its hierarchical (HIER) extension (Libovický and Helcl, 2017).”
Evaluating Discourse Phenomena in Neural Machine Translation (2017, methodology): “Inspired by work on multi-modal translation (Caglayan et al., 2016; Huang et al., 2016), multi-encoder translation models have recently been used to incorporate extra-sentential linguistic context in purely textual NMT (Zoph and Knight, 2016; Libovick´y and Helcl, 2017; Wang et al., 2017).”
An Exploration of Neural Sequence-to-Sequence Architectures for Automatic Post-Editing (2017, unknown): “Although applications to the APE problem have been reported (Libovický and Helcl, 2017), state-of-the-art results seem to be missing.”
Vision Guided Generative Pre-trained Language Models for Multimodal Abstractive Summarization (2021, background): “attention (HA) (Libovickỳ and Helcl, 2017) that can integrates information from different modalities into a coherent output.”
Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Description (2017, methodology): “It uses a hierarchical attention combination in the decoder (Libovický and Helcl, 2017).”
How2: A Large-scale Dataset for Multimodal Language Understanding (2018, methodology): “For multimodal summarization, we follow the hierarchical attention approach [17, 18] to combine textual and visual modalities by using a sequence of action-level features instead of an average-pooled one as in the other experiments.”
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data (2020, methodology): “At each time step, an LSTM decoder performs hierarchical attention (Libovický and Helcl, 2017) over the list of table-specific representations, constructing an”
A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets (2021, unknown): “They used hierarchical attention [114] as a merging mechanism that can be used to generate a synthesis vector frommultimodal sources.”
Multi-Source Transformer for Kazakh-Russian-English Neural Machine Translation (2019, methodology): “Training a multi-encoder (Libovick´y and Helcl, 2017; Libovick´y et al., 2018) Trans-former system (Vaswani et al., 2017) from Kazakh/Russian to English that subsumes both of these approaches (Fig.”
On Vision Features in Multimodal Machine Translation (2022, background): “Early models of this kind produce a translation given the fused representation of both the visual and textual inputs (Caglayan et al., 2016; Libovický and Helcl, 2017; Calixto and Liu, 2017).”
QuickEdit: Editing Text & Translations via Simple Delete Actions (2017, background): “Dual attention has been introduced recently in the context of automatic post-editing (Novak et al., 2016; Libovickỳ and Helcl, 2017).”
A Shared Attention Mechanism for Interpretation of Neural Automatic Post-Editing Systems (2018, methodology): “Note that our model differs from that proposed by (Libovickỳ and Helcl, 2017) only in that we do not employ the learnable projection matrices, Uc(k).”
Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation (2021, methodology): “This visual representation can be used to initialize the encoder/decoder’s hidden vectors (Elliott et al., 2015; Libovický and Helcl, 2017; Calixto et al., 2016).”
QuickEdit: Editing Text & Translations by Crossing Words Out (2017, background): “Dual attention has been introduced recently in the context of automatic post-editing (Novak et al., 2016; Libovickỳ and Helcl, 2017).”