Grounding Action Descriptions in Videos

By Michaela Regneri, Marcus Rohrbach, Dominikus Wetzel, Stefan Thater, Bernt Schiele, Manfred Pinkal. 2013

568 citations recorded by the source corpus.

Recent work has shown that the integration of visual information into text-based models can substantially improve model predictions, but so far only visual information extracted from static images has been used. In this paper, we consider the problem of grounding sentences describing actions in visual information extracted from videos. We present a general purpose corpus that aligns high quality videos with multiple natural language descriptions of the actions portrayed in the videos, together with an annotation of how similar the action descriptions are to each other. Experimental results demonstrate that a text-based model of similarity between actions improves substantially when combined with visual information from videos depicting the described actions.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

Experience Grounds Language (2020, background): “Wang et al. (2019b) extends this paradigm by exploring machine translation with videos serving as an interlingua, and Regneri et al. (2013) introduced a foundational dataset which aligns text descriptions and semantic annotations of actions with videos.”
ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks (2019, neutral): “and concepts [1, 46, 43, 56, 59].”
Unsupervised Learning from Narrated Instruction Videos (2015, background): “proach. Relatively little work exploits vision to improve NLP or to ground linguistic expressions. [17] improves coreference resolution by learning appearance of people in video. Similar to our work, [20] demonstrates that a text-based model of similarity between actions improves when combined with ”
Dense-Captioning Events in Videos (2017, methodology): “TT [55], MSVD [6] and YouCook [7] collected a dataset with slightly longer length, at the cost of a few descriptions than previous datasets. To further improve video annotations, KITTI [12] and TACoS [36] also temporally localized their video descriptions. Orthogonally, in an effort to increase the ”
Movie Description (2016, background): “2013) x cooking crowd 88 - 2,668 TACoS (Regneri et al. 2013) x cooking crowd 127 7,206 18,227 TACoS Multi-Level (Rohrbach et al.”
A dataset for Movie Description (2015, background): “We show competitive performance on the TACoS Multi-Level dataset and promising results on our movie description data.”
Localizing Moments in Video with Natural Language (2017, background): “Additionally, datasets like [20, 28] are small and restricted to spe-ciﬁc domains, such as dash-cam or cooking videos, while datasets [5, 32, 45, 50] sourced from movies and YouTube are frequently edited and tend to only include entertaining moments (see [38] for discussion).”
Grounding of Textual Phrases in Images by Reconstruction (2015, background): “[27] aims to understand similar action language descriptions by associating them with corresponding videos, but they do not localize the descriptions spatially or temporally.”
TGIF: A New Dataset and Benchmark on Animated GIF Description (2016, methodology): “The TACoS dataset [30] contains 9K cooking video clips and 12K descriptions, while the YouCook dataset [8] contains 80 cooking videos and 748 descriptions.”
The “Something Something” Video Database for Learning and Evaluating Visual Common Sense (2017, neutral): “Many shortcomings of existing video datasets may be re-Table [26] action (cooking) 44 600s - TACoS [23] action (cooking) 127 360s - Charades [29] action (human) 10, 000 30s - KITTI [6] action (driving) 21 30s - Something-Something (ours) human-object interaction 108,499 4.03s 174 ﬁne-grained…”
A Dataset and Exploration of Models for Understanding Video Data through Fill-in-the-Blank Question-Answering (2016, methodology): “Their data is created by reformulating various video description datasets including TACoS [26], MPII-MD [29] and the TRECVID MEDTest 14 [1] dataset.”
Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation (2017, background): “There is a growing body of work that generalises the task from static inputs to sequential ones, especially videos (e.g. Kojima et al., 2002; Regneri et al., 2013; Venugopalan et al., 2015b, 2015a).”
MarioQA: Answering Questions by Watching Gameplay Videos (2016, methodology): “[29] constructed a VideoQA dataset in three domains using existing videos with grounded descriptions, where the domains include cooking scenarios [11], movie clips [12] and web videos [1].”
Visual Relationship Detection with Language Priors (2016, background): “ general as object 1 is not constrained to be a human and the predicate doesn’t have to be a verb. Visual relationships are not a new concept. Some papers explicitly collected relationships in images [25,26,27,28,29] and videos [27,30,31] and helped models map these relationships from images to lang”
Weakly-Supervised Alignment of Video with Text (2015, methodology): “This representation makes sense on the TACoS dataset as sentences are in general pretty simple.”
TALL: Temporal Activity Localization via Language Query (2017, neutral): “TACoS [22].”