Unsupervised Visual-Linguistic Reference Resolution in Instructional Videos

By De-An Huang, Joseph J. Lim, Li Fei-Fei, Juan Carlos Niebles. 2017

57 citations recorded by the source corpus.

We propose an unsupervised method for reference resolution in instructional videos, where the goal is to temporally link an entity (e.g., dressing) to the action (e.g., mix yogurt) that produced it. The key challenge is the inevitable visual-linguistic ambiguities arising from the changes in both visual appearance and referring expression of an entity in the video. This challenge is amplified by the fact that we aim to resolve references with no supervision. We address these challenges by learning a joint visual-linguistic model, where linguistic cues can help resolve visual ambiguities and vice versa. We verify our approach by learning our model unsupervisedly using more than two thousand unstructured cooking videos from YouTube, and show that our visual-linguistic model can substantially improve upon state-of-the-art linguistic only model on reference resolution in instructional videos.

Subfields: Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips (2019, background): “Instructional videos are rising in popularity in the context of learning steps of complex tasks [2, 16, 41, 42, 46, 68], visual-linguistic reference resolution [17, 18], action segmentation in long untrimmed videos [66] and joint learning of object states and actions [3].”
MERLOT: Multimodal Neural Script Knowledge Models (2021, methodology): “Prior works have used web videos with ASR to build weakly-supervised object detectors [87], action detectors/classiﬁers [120, 6, 62, 84], instruction aligners [77, 5, 19], video captioners [96, 46, 86, 101], and visual reference resolvers [49].”
Localizing Moments in Video with Natural Language (2017, background): “ 2 Related Work Localizing Video Segments with Natural Language. Prior work has considered aligning natural language with video, e.g., instructional videos with transcribed text (Kiddon et al., 2015; Huang et al., 2017; Malmaud et al., 2014, 2015). Our work is most related to recent work in video mo”
Cross-Task Weakly Supervised Learning From Instructional Videos (2019, background): “These have attracted a great deal of attention recently [1, 2, 19, 20, 24, 29, 35] since the co-occurrence of demonstrative visual actions and natural language enables many interesting tasks ranging from coreference resolution [19] to learning person-object interaction [2, 10].”
COIN: A Large-Scale Dataset for Comprehensive Instructional Video Analysis (2019, background): “Also, growing efforts have been devoted to exploiting different challenges of instructional video analysis [10, 24, 36, 52].”
AVLnet: Learning Audio-Visual Language Representations from Instructional Videos (2020, methodology): “A variety of tasks, focused primarily on text-video modelling, have been applied to these datasets including: task segmentation [19–21], reference resolution [22, 23], action segmentation [5], video clip ordering [24–27], and action recognition [28–30].”
Finding "It": Weakly-Supervised Reference-Aware Visual Grounding in Instructional Videos (2018, methodology): “We initialize the reference edges R by unsupervised reference resolution from [16].”
Neural Task Graphs: Generalizing to Unseen Tasks From a Single Video Demonstration (2018, unknown): “This includes instructional video transcript [32,33,34], movie script [35,36], and caption annotation [37,38].”
Reward Learning from Narrated Demonstrations (2018, methodology): “The videos in our dataset are instructional in nature, similar to instructional videos on YouTube [26, 3].”
Visual Reference Resolution using Attention Memory for Visual Dialog (2017, background): “This progress, in turn, has enabled a variety of multi-modal applications spanning both domains, including image captioning [2–4], language grounding [5, 6], image generation from captions [7, 8], and visual question answering (VQA) on images [9–21] and videos [22–24].”
Action Modifiers: Learning From Adverbs in Instructional Videos (2019, methodology): “Narrations are used to learn steps of complex tasks [1, 18, 30, 42, 45, 62], and more recently for video retrieval [33], visual grounding [17, 19], action segmentation [60] and learning actions through object state changes [2, 14].”