Finding "It": Weakly-Supervised Reference-Aware Visual Grounding in Instructional Videos

By De-An Huang, Shyamal Buch, Lucio M. Dery, Animesh Garg, Li Fei-Fei, Juan Carlos Niebles. 2018

103 citations recorded by the source corpus.

Grounding textual phrases in visual content with standalone image-sentence pairs is a challenging task. When we consider grounding in instructional videos, this problem becomes profoundly more complex: the latent temporal structure of instructional videos breaks independence assumptions and necessitates contextual understanding for resolving ambiguous visual-linguistic cues. Furthermore, dense annotations and video data scale mean supervised approaches are prohibitively costly. In this work, we propose to tackle this new task with a weakly-supervised framework for reference-aware visual grounding in instructional videos, where only the temporal alignment between the transcription and the video segment are available for supervision. We introduce the visually grounded action graph, a structured representation capturing the latent dependency between grounding and references in video. For optimization, we propose a new reference-aware multiple instance learning (RA-MIL) objective for weak supervision of grounding in videos. We evaluate our approach over unconstrained videos from YouCookII and RoboWatch, augmented with new reference-grounding test set annotations. We demonstrate that our jointly optimized, reference-aware approach simultaneously improves visual grounding, reference-resolution, and generalization to unseen instructional video categories.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips (2019, background): “Instructional videos are rising in popularity in the context of learning steps of complex tasks [2, 16, 41, 42, 46, 68], visual-linguistic reference resolution [17, 18], action segmentation in long untrimmed videos [66] and joint learning of object states and actions [3].”
COIN: A Large-Scale Dataset for Comprehensive Instructional Video Analysis (2019, methodology): “, step localization, action segmentation, procedure segmentation [52], dense video caption [53] and visual grounding [25, 51].”
AVLnet: Learning Audio-Visual Language Representations from Instructional Videos (2020, methodology): “A variety of tasks, focused primarily on text-video modelling, have been applied to these datasets including: task segmentation [19–21], reference resolution [22, 23], action segmentation [5], video clip ordering [24–27], and action recognition [28–30].”
Visual Relation Grounding in Videos (2020, background): “rts are made on referring expression grounding in static images [8,19,22,23,28,41,42,44], recent research attempts to study visual grounding in videos by nding the objects either in individual frames [10,32,47] or in video clips spatio-temporally [1,3,46]. Nonetheless, all these works focus on groun”
Grounding Object Detections With Transcriptions (2019, methodology): “Shi et al. (2019); Zhou et al. (2019; 2018a); Huang et al. (2018); Zhao et al. (2018); Ephrat et al. (2018) propose approaches on grounding speech and textual representations to videos.”
Neural Task Graphs: Generalizing to Unseen Tasks From a Single Video Demonstration (2018, unknown): “This includes instructional video [2, 18, 34], movie script [37, 45], and caption annotation [14, 25].”
D3TW: Discriminative Differentiable Dynamic Time Warping for Weakly Supervised Action Alignment and Segmentation (2019, background): “The supervision usually comes from movie scripts [8, 2, 41] or transcription of instructional videos [1, 33, 25, 14].”
Procedure Planning in Instructional Videos (2019, background): “This opens the opportunities of addressing challenging tasks on real-world videos, including temporal segmentation and captioning [20, 26], state understanding [3], and reference resolution [10].”
Recursive Visual Attention in Visual Dialog (2018, methodology): “Co-reference resolution has been used to improve visual comprehension in many tasks, such as visual grounding [13], action recognition [25, 26] and scene understanding [16].”
Injecting Semantic Concepts into End-to-End Image Captioning (2021, methodology): “This indicates the potential application of ViCAP for weakly supervised textual grounding tasks for the image [16, 17, 59, 68] and video [26, 50].”
Action Modifiers: Learning From Adverbs in Instructional Videos (2019, methodology): “Narrations are used to learn steps of complex tasks [1, 18, 30, 42, 45, 62], and more recently for video retrieval [33], visual grounding [17, 19], action segmentation [60] and learning actions through object state changes [2, 14].”