Describing Multimedia Content Using Attention-Based Encoder-Decoder Networks

By Kyunghyun Cho, Aaron C. Courville, Yoshua Bengio. 2015

442 citations recorded by the source corpus.

Whereas deep neural networks were first mostly used for classification tasks, they are rapidly expanding in the realm of structured output problems, where the observed target is composed of multiple random variables that have a rich joint distribution, given the input. In this paper we focus on the case where the input also has a rich structure and the input and output structures are somehow related. We describe systems that learn to attend to different places in the input, for each element of the output, for a variety of tasks: machine translation, image caption generation, video clip description, and speech recognition. All these systems are based on a shared set of building blocks: gated recurrent neural networks and convolutional neural networks, along with trained attention mechanisms. We report on experimental results with these systems, showing impressively good performance and the advantage of the attention mechanism.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Model Architectures

Citations in the corpus

Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering (2015, methodology): “[19][20][21][23] use the “concatenation” alignment function in their soft attention models and [24] gives a literature review of such models applied to different tasks.”
DenseCap: Fully Convolutional Localization Networks for Dense Captioning (2015, methodology): “Several recent approaches to Image Captioning [32, 21, 49, 8, 4, 24, 11] rely on a combination of RNN language model conditioned on image information, possibly with soft attention mechanisms [51, 5].”
Phased LSTM: Accelerating Recurrent Network Training for Long or Event-based Sequences (2016, background): “Applications include speech recognition [13], natural language processing [1, 20], and attention-based models for structured prediction [5, 29].”
Using deep learning to detect price change indications in financial markets (2017, background): “Also, an attention mechanism [6], [19], can be introduced to allow the network to capture only the relevant information and avoid noise.”
Pose-conditioned Spatio-Temporal Attention for Human Action Recognition (2017, background): “Recently, soft attention was proposed for image [7] and video understanding [33, 34, 42], with spatial, temporal and spatiotemporal variants.”
Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement (2018, methodology): “, Cho et al., 2015, and references therein). Much of this recent success is built on top of successful autoregressive sequence modeling in which the probability of a target sequence is factorized as a product of conditional probabilities of next symbols given all the preceding ones. Despite its succ”
Insertion-based Decoding with Automatically Inferred Generation Order (2019, background): “, the L2R order is proven to be a strong baseline in many scenarios (Cho et al., 2015), we assume a Diracdelta distribution q(π|X,Y ) = δ(π = π∗(X,Y )), where π∗(X,Y ) is a predefined order given X and Y .”
Search Engine Guided Non-Parametric Neural Machine Translation (2017, methodology): “By embedding an input of any modality into a fixed vector space and using fast approximate search (Douze et al., 2017), this approach can, for instance, be used for multimedia description generation (Cho et al., 2015).”
Non-Monotonic Sequential Text Generation (2019, background): “It allows us to build a conditional non-monotonic sequence generator that can for instance be used for machine translation, image caption generation, speech recognition and generally multimedia description generation (Cho et al., 2015).”
X-Linear Attention Networks for Image Captioning (2020, background): “After that, a series of innovations have been proposed to boost image captioning by encouraging more interactions between the two different modalities via attention mechanism [5].”
Latent Alignment and Variational Attention (2018, background): “Neural soft attention models were originally introduced as an alternative approach for neural machine translation [6], and have subsequently been successful on a wide range of tasks (see [15] for a review of applications).”
Explaining Therapy Predictions with Layer-Wise Relevance Propagation in Neural Networks (2018, background): “Especially in image data and language data, this approach has been demonstrating promising explainability in combination with CNNs and RNNs, respectively.”