Weakly-Supervised Video Object Grounding from Text by Loss Weighting and Object Interaction

By Luowei Zhou, Nathan Louis, Jason J. Corso. 2018

105 citations recorded by the source corpus.

We study weakly-supervised video object grounding: given a video segment and a corresponding descriptive sentence, the goal is to localize objects that are mentioned from the sentence in the video. During training, no object bounding boxes are available, but the set of possible objects to be grounded is known beforehand. Existing approaches in the image domain use Multiple Instance Learning (MIL) to ground objects by enforcing matches between visual and semantic features. A naive extension of this approach to the video domain is to treat the entire segment as a bag of spatial object proposals. However, an object existing sparsely across multiple frames might not be detected completely since successfully spotting it from one single frame would trigger a satisfactory match. To this end, we propagate the weak supervisory signal from the segment level to frames that likely contain the target object. For frames that are unlikely to contain the target objects, we use an alternative penalty loss. We also leverage the interactions among objects as a textual guide for the grounding. We evaluate our model on the newly-collected benchmark YouCook2-BoundingBox and show improvements over competitive baselines.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

VideoBERT: A Joint Model for Video and Language Representation Learning (2019, methodology): “rb and noun labels from the tokens predicted in the ﬁrst and second masked slots, respectively. See Figure5for some qualitative results. For quantitative evaluation, we use the YouCook II dataset. In [37], the authors collected ground truth bounding boxes for the 63 most common objects for the valid”
Contrastive Bidirectional Transformer for Temporal Representation Learning (2019, background): “Language can be leveraged by finding a joint embedding space for both visual and textual modalities or by learning an alignment between the two modalities [14, 2, 38].”
Grounded Video Description (2018, background): “in the tables) over GT sentences following [25, 40].”
AVLnet: Learning Audio-Visual Language Representations from Instructional Videos (2020, methodology): “Each audio embedding is mapped to the ASR-detected words in the surrounding 2 seconds, and each visual embedding is mapped to a set of food labels provided by YouCook2 [46].”
Visual Relation Grounding in Videos (2020, background): “While these works focus on image grounding, more recent efforts [1,3,10,32,39,46,47] also attempted to ground objects in videos.”
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives (2023, unknown): “Research in instructional or “how-to” videos is facilitated by (largely exo) datasets like HowTo100M [103] and others [11, 159, 204, 207].”