Instructional Videos for Unsupervised Harvesting and Learning of Action Examples

By Shoou-I Yu, Lu Jiang, Alexander G. Hauptmann. 2014

62 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning Video Representations from Textual Web Supervision (2020, background): “Prior works have leveraged ambient sound [52, 5, 4, 71, 34, 51, 55], dialogue [49], and narration [70, 2, 72, 73, 46, 45, 3], all of which of which serve as useful signals.”
Self-Supervised MultiModal Versatile Networks (2020, background): “Recently, instructional videos became a popular source of video and language data [6, 51, 75, 94] due to not requiring careful manual annotation, e.”
End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, positive): “To avoid labeling visual data, several approaches have leveraged audio transcripts obtained from narrated videos using automatic speech recognition (ASR) as a way to supervise video models for object detection [4, 16, 56], captioning [34, 72], classification [3, 44, 49, 89], summarization [59] or re”
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips (2019, background): “Related to our work, [2, 30, 62] also consider automatically generated transcription of narrated instructional videos as a source of supervision.”
MERLOT: Multimodal Neural Script Knowledge Models (2021, methodology): “Prior works have used web videos+ASR to build weakly-supervised object detectors [75], action detectors/classifiers [108, 5, 51, 72], instruction aligners [65, 4, 15], video captioners [85, 38, 74, 89], and visual reference resolvers [41].”
Connectionist Temporal Modeling for Weakly Supervised Action Labeling (2016, background): “An alternative is to develop methods that require weak supervision, which may be automatically extracted from movie scripts [3,5,9] or instructional videos [2,4,50] at a lower cost.”
Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models (2021, background): “Prior work has shown that instructional videos are useful for event recognition (Yu et al., 2014), action localization model (Alayrac et al., 2016), cross-modal alignments (Malmaud et al., 2015).”
Grounding Object Detections With Transcriptions (2019, methodology): “For this reason, to overcome this difficulty, there has been numerous works on weakly-supervised models for object and event classification (Zhuang et al., 2017; Guo et al., 2018; Yu et al., 2014; Chang et al., 2015).”
Self-Paced Curriculum Learning (2015, background): “…features (Tan, Jiang, and Neo 2014; Liu et al. 2013; Zhang et al. 2014; Inoue and Shinoda 2014; Jiang, Hauptmann, and Xiang 2012; Tang et al. 2012; Yu, Jiang, and Hauptmann 2014; Cao et al. 2013), and by automatic speech recognition (Miao, Metze, and Rawat 2013; Miao et al. 2014; Chiu and…”
Visual Grounding in Video for Unsupervised Word Translation (2020, methodology): “In this work, we rely on instructional videos [1, 40, 49] since they can be obtained at scale without any manual annotation [35]: they consist of YouTube videos and their associated narrations which is generated using automatic speech recognition (ASR).”
How2: A Large-scale Dataset for Multimodal Language Understanding (2018, methodology): “[42] Shoou-I Yu, Lu Jiang, and Alexander Hauptmann.”
Finding "It": Weakly-Supervised Reference-Aware Visual Grounding in Instructional Videos (2018, methodology): “This use of transcription as supervision has been utilized in several contexts, such as action detection [55], object state discovery [2], entity reference [16], and procedural knowledge discovery [1, 29, 45].”
Learning Temporal Dynamics from Cycles in Narrated Video (2021, background): “Other work instead learns from automatic transcripts of narrations in instructional videos [1, 33, 63].”
Beyond Instructional Videos: Probing for More Diverse Visual-Textual Grounding on YouTube (2020, unknown): “…video understanding tasks (Gupta et al., 2017; Huang et al., 2017; Huang* et al., 2018; Moriya et al., 2019), e.g., action detection/classiﬁcation (Yu et al., 2014; Alayrac et al., 2017; Chang et al., 2019; Kuehne et al., 2019), segmentation/captioning (Sener et al., 2015), and instruction…”
Revealing Event Saliency in Unconstrained Video Collection (2017, unknown): “The toolkit builds the HMM (Hidden Markov Model)/GMM (Gaussian Mixture Model) acoustic model with speaker adaptive training, where the models are trained from instructional video data [34].”