Grounding Object Detections With Transcriptions

By Yasufumi Moriya, Ramon Sanabria, Florian Metze, Gareth J. F. Jones. 2019

6 citations recorded by the source corpus.

A vast amount of audio-visual data is available on the Internet thanks to video streaming services, to which users upload their content. However, there are difficulties in exploiting available data for supervised statistical models due to the lack of labels. Unfortunately, generating labels for such amount of data through human annotation can be expensive, time-consuming and prone to annotation errors. In this paper, we propose a method to automatically extract entity-video frame pairs from a collection of instruction videos by using speech transcriptions and videos. We conduct experiments on image recognition and visual grounding tasks on the automatically constructed entity-video frame dataset of How2. The models will be evaluated on new manually annotated portion of How2 dev5 and val set and on the Flickr30k dataset. This work constitutes a first step towards meta-algorithms capable of automatically construct task-specific training sets.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, methodology): “To avoid labeling visual data, several approaches have leveraged audio transcripts obtained from narrated videos using automatic speech recognition (ASR) as a way to supervise video models for object detection [4, 16, 56], captioning [34, 72], classification [3, 44, 49, 89], summarization [59] or re”
MERLOT: Multimodal Neural Script Knowledge Models (2021, methodology): “Prior works have used web videos+ASR to build weakly-supervised object detectors [75], action detectors/classifiers [108, 5, 51, 72], instruction aligners [65, 4, 15], video captioners [85, 38, 74, 89], and visual reference resolvers [41].”
Beyond Instructional Videos: Probing for More Diverse Visual-Textual Grounding on YouTube (2020, background): “…known to be a useful signal source in various instructional video understanding tasks (Gupta et al., 2017; Huang et al., 2017; Huang* et al., 2018; Moriya et al., 2019), e.g., action detection/classiﬁcation (Yu et al., 2014; Alayrac et al., 2017; Chang et al., 2019; Kuehne et al., 2019),…”