Movie/Script: Alignment and Parsing of Video and Text Transcription

By Timothée Cour, Christopher T. Jordan, Eleni Miltsakaki, B. Taskar. 2008

149 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Natural Language Processing

Citations in the corpus

Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books (2015, background): “Early work on movie-to-text alignment include dynamic time warping for aligning movies to scripts with the help of subtitles [6, 5].”
MovieQA: Understanding Stories in Movies through Question-Answering (2015, background): “In [7, 34], the authors aligned videos with movie scripts in order to improve scene prediction.”
Movie Description (2016, methodology): “We follow existing approaches (Cour et al. 2008; Laptev et al. 2008) to automatically align scripts to movies.”
A dataset for Movie Description (2015, methodology): “Second, we propose to use semantic parsing of the sentence to extract training labels for recently proposed translation approach [56] for video description.”
Grounding Action Descriptions in Videos (2013, background): “…far been mostly explored by the computer vision community, where different methods for improving action recognition by exploiting linguistic data have been proposed (Gupta and Mooney, 2010; Motwani and Mooney, 2012; Cour et al., 2008; Tzoukermann et al., 2011; Rohrbach et al., 2012b, among others).”
Multimodal Machine Learning: A Survey and Taxonomy (2017, background): “[46] to align movie shots and scenes to the corresponding screenplay.”
Ava Active Speaker: An Audio-Visual Dataset for Active Speaker Detection (2019, background): “…has multiple applications, such as in interactive systems that identify the speaker and per-sonalize responses [44, 45], in speech transcription, speaker diarization and speech enhancement systems [1, 9, 12, 26, 41, 42], and for tracking storylines and characters in narrative content [10, 14].”
Subtitle-free Movie to Script Alignment (2009, background): “Scripts aligned with videos offer a number of possibilities: they can provide supervisory information for identifying characters [7] or learning actions [13]; they enable a scene level organization of the video material [6]; and they enable text-based retrieval and search [16].”
Learning from Partial Labels (2011, methodology): “…on video, a wide range of cues has been used to automatically obtain supervised data, including: captions or transcripts (Everingham et al., 2006; Cour et al., 2008; Laptev et al., 2008), sound (Satoh et al., 1999) to obtain the transcript, or clustering based on clothing, face and hair color…”
Automatic annotation of human actions in video (2009, methodology): “Our automatic video annotation is based on video alignment of scripts used in [5, 8, 13].”
“Who are you?” - Learning person specific classifiers from video (2009, background): “Note that our aims differ from related work which has made more extensive use of script text [6] in that we aim to build models which allow labelling of faces with no associated text, rather than selecting from candidate names obtained from a full transcript.”
Learning from ambiguously labeled images (2009, background): “presented in [4, 7], linking closed captions to screenplay.”
Actions in context (2009, methodology): “Scripts have been explored as a means of automatic video annotation in [2, 5, 12].”
Label Ranking under Ambiguous Supervision for Learning Semantic Correspondences (2010, background): “For instance, the time synchronisation of several media can generate annotated corpora: matching movies with corresponding scripts can be used for speech recognition or information retrieval in videos (Cour et al., 2008), matching vision sensors and other sensors can be used to improve robotic visio”
Video Event Understanding Using Natural Language Descriptions (2013, unknown): “Previous works like [13, 16, 4] use time-synchronized movie scripts or closed captions to identify video segments corresponding to specific actions.”
Modeling the Temporal Extent of Actions (2010, background): “In [1], [2], [3] and [4] the authors leverage the availability of movie scripts and closed captioning to get a rough idea of when actions occur in movies or television shows.”