YouTube2Text: Recognizing and Describing Arbitrary Activities Using Semantic Hierarchies and Zero-Shot Recognition

By Sergio Guadarrama, Niveda Krishnamoorthy, Girish Malkarnenkar, Subhashini Venugopalan, Raymond J. Mooney, Trevor Darrell, Kate Saenko. 2013

505 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Natural Language Generation

Citations in the corpus

NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions (2021, background): “where WUP (p, r) calculates the Wu-Parlmer similarity [2, 8] of two words based on their depth in the taxonomy [1, 6]: WUP(p, r) = 2*depth(lcs) / (depth(p) + depth(r)), in which lcs is the least common ancestor of the words p and r.”
Hero: Hierarchical Encoder for Video+Language Omni-representation Pre-training (2020, background): “, 2018b, 2019), mostly benchmarking on Youtube2Text (Guadarrama et al., 2013), MSR-VTT (Xu et al.”
VQA: Visual Question Answering (2015, methodology): “Related to VQA are the tasks of image tagging [8, 23], image captioning [24, 14, 34, 7, 13, 45, 9, 19, 32, 21] and video captioning [39, 17], where words or sentences are generated to describe visual content.”
Long-term recurrent convolutional networks for visual recognition and description (2014, methodology): “[11, 30, 17, 3, 6, 17, 41, 42] propose methods for generating sentence descriptions for video, but to our knowledge we present the first application of deep models to the video description task.”
ReferItGame: Referring to Objects in Photographs of Natural Scenes (2014, background): “Very recently, these ideas have been extended to produce descriptions for videos (Guadarrama et al., 2013; Barbu et al., 2012).”
A Joint Sequence Fusion Model for Video Question Answering and Retrieval (2018, background): “Recently, various video-language tasks have drawn a lot of interests in computer vision research [1,2,3], including video captioning [4,5,6,7,8,9], video question answering (QA) [10,11], and video retrieval for a natural language query [8,12,13].”
A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Input (2014, methodology): “For μ we use a variant of Wu-Palmer similarity [23, 24].”
Movie Description (2016, background): “In the past video description has been addressed in controlled settings (Barbu et al. 2012; Kojima et al. 2002), on a small scale (Das et al. 2013; Guadarrama et al. 2013; Thomason et al. 2014) or in single domains like cooking (Rohrbach et al. 2014, 2013; Don-ahue et al. 2015).”
Generating Visual Explanations (2016, methodology): “Early image description methods rely on ﬁrst detecting visual concepts in a scene (e.g., subject, verb, and object) before generating a sentence with either a simple language model or sentence template [23,24].”
Sequence to Sequence -- Video to Text (2015, methodology): “Several previous methods for generating sentence descriptions [11, 19, 36] used a two stage pipeline that first identifies the semantic content (subject, verb, object) and then generates a sentence based on a template.”
Translating Videos to Natural Language Using Deep Recurrent Neural Networks (2014, methodology): “The previous work on the YouTube corpus we employ (Motwani and Mooney, 2012; Krishnamoorthy et al., 2013; Guadarrama et al., 2013; Thomason et al., 2014) used a two-step approach, first detecting a fixed tuple of role words, such as subject, verb, object, and scene, and then using a template to gene”
TGIF: A New Dataset and Benchmark on Animated GIF Description (2016, methodology): “In the video domain, the YouTube2Text dataset [5, 11] contains 2K video clips and 120K sentences.”
CLEVRER: CoLlision Events for Video REpresentation and Reasoning (2019, background): “This includes video captioning (Guadarrama et al., 2013; Venugopalan et al., 2015; Gan et al., 2017), localizing video segments from natural language queries (Gao et al., 2017; Hendricks et al., 2017), and video question answering.”
Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks (2015, methodology): “To eliminate the tedious effort of rule engineering when the problem scales, some recent methods train statistical models for lexical entries, either in a fully [10, 14, 24, 41] or weakly [36, 35, 57, 54] supervised fashion.”
Jointly Modeling Embedding and Translation to Bridge Video and Language (2015, background): “Guadarrama et al. [11] use semantic hierarchies to choose an appropriate level of the specificity and accuracy of sentence fragments.”
Using Descriptive Video Services to Create a Large Data Source for Video Annotation Research (2015, background): “The majority of available video datasets are toy domains with with small word vo-cabularies such as TaCos [17], TaCos Multi-Level [18], YouCook [11], and MSVD [9].”
Coherent Multi-sentence Video Description with Variable Level of Detail (2014, background): “Recently it has received increased interest both in the computer vision [1, 3, 4, 10, 14] and computational linguistic communities [9, 11, 22].”