Translating Video Content to Natural Language Descriptions

By Marcus Rohrbach, Wei Qiu, Ivan Titov, Stefan Thater, Manfred Pinkal, Bernt Schiele. 2013

381 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models (2014, methodology): “This point of view has also been used by [28] and allows us to make use of existing ideas in the machine translation literature.”
VQA: Visual Question Answering (2015, methodology): “Related to VQA are the tasks of image tagging [8, 23], image captioning [24, 14, 34, 7, 13, 45, 9, 19, 32, 21] and video captioning [39, 17], where words or sentences are generated to describe visual content.”
Long-term recurrent convolutional networks for visual recognition and description (2014, methodology): “The results show that (1) the LSTM outperforms an SMT-based approach to video description; (2) the simpler decoder architecture (b) and (c) achieve better performance than (a), likely because the input does not need to be memorized; and (3) our approach achieves 28.8%, clearly outperforming the best”
CIDEr: Consensus-based image description evaluation (2014, methodology): “Most of the approaches use the UIUC Pascal Sentence [14, 22, 29, 37, 17] and the MS COCO datasets [19, 6, 10, 43, 27, 21] for evaluation.”
Visual7W: Grounded Question Answering in Images (2015, background): “Image and video captioning has become a popular task in the past year [4, 5, 13, 37, 45, 47].”
MovieQA: Understanding Stories in Movies through Question-Answering (2015, methodology): “In [10, 41], the authors caption video clips using LSTMs, [33] formulates description as a machine translation model, while older work uses templates [3, 8, 18].”
A dataset for Movie Description (2015, methodology): “To be able to learn how to generate descriptions of visual content, parallel datasets of visual content paired with descriptions are indispensable [56].”
Sequence to Sequence -- Video to Text (2015, background): “Describing visual content with natural language text has recently received increased interest, especially describing images with a single sentence [8, 5, 16, 18, 20, 23, 29, 40].”
Translating Videos to Natural Language Using Deep Recurrent Neural Networks (2014, background): “Solutions have been proposed for narrow domains with a small set of known actions and objects, e.g., (Barbu et al., 2012; Rohrbach et al., 2013), but generating descriptions for “in-the-wild” videos such as the YouTube domain (Figure 1) remains an open challenge.”
Localizing Moments in Video with Natural Language (2017, background): “TACoS [28, 34] 123/7,206 18,227 Lab Kitchen Cooking X X”
Describing Videos by Exploiting Temporal Structure (2015, background): “Video description generation has been investigated and studied in other work, such as [19, 2, 26].”
TGIF: A New Dataset and Benchmark on Animated GIF Description (2016, methodology): “[ 32] formulate video description as a translation problem and propose a method that combines semantic role labeling and statistical machine translation.”
Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks (2015, methodology): “To eliminate the tedious effort of rule engineering when the problem scales, some recent methods train statistical models for lexical entries, either in a fully [10, 14, 24, 41] or weakly [36, 35, 57, 54] supervised fashion.”
Jointly Modeling Embedding and Translation to Bridge Video and Language (2015, methodology): “For video translation, Rohrbach et al. [23] learn a CRF to model the relationships between different components of the input video and generate descriptions for video.”
Using Descriptive Video Services to Create a Large Data Source for Video Annotation Research (2015, background): “The majority of available video datasets are toy domains with with small word vo-cabularies such as TaCos [17], TaCos Multi-Level [18], YouCook [11], and MSVD [9].”
Jointly Modeling Deep Video and Compositional Text to Bridge Vision and Language in a Unified Framework (2015, background): “" (Rohrbach et al. 2013) propose a CRF to model relationships between different visual components and translate video to text in cooking videos.”
Coherent Multi-sentence Video Description with Variable Level of Detail (2014, methodology): “As mentioned above, the approach of [19] uses visual attributes to obtain the features for CRF unaries.”
A Dataset and Exploration of Models for Understanding Video Data through Fill-in-the-Blank Question-Answering (2016, background): “Early models tackling video captioning such as [17, 32], focused on constrained domains with limited appearance of activities and objects in videos and depended heavily on hand-crafted video features, followed by a template-based or shallow statistical machine translation.”