Finding Actors and Actions in Movies

By Piotr Bojanowski, Francis R. Bach, Ivan Laptev, Jean Ponce, Cordelia Schmid, Josef Sivic. 2013

152 citations recorded by the source corpus.

We address the problem of learning a joint model of actors and actions in movies using weak supervision provided by scripts. Specifically, we extract actor/action pairs from the script and use them as constraints in a discriminative clustering framework. The corresponding optimization problem is formulated as a quadratic program under linear constraints. People in video are represented by automatically extracted and tracked faces together with corresponding motion features. First, we apply the proposed framework to the task of learning names of characters in the movie and demonstrate significant improvements over previous methods used for this task. Second, we explore the joint actor/action constraint and show its advantage for weakly supervised action learning. We validate our method in the challenging setting of localizing and recognizing characters and their actions in feature length movies Casablanca and American Beauty.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, neutral): “Multiple instance learning methods have been employed in many weakly-supervised video understanding problems including: person recognition in movies using scripts [12, 50, 61], anomaly detection [70], weakly supervised action classification [47, 68] and localization [17, 22, 82], co-reference resolu”
Joint Discovery of Object States and Manipulation Actions (2017, methodology): “In particular, we build on the discriminative clustering approach of [4] that has been shown to perform well in a variety of computer vision problems [5].”
Unsupervised Learning from Narrated Instruction Videos (2015, background): “More recently, weakly supervised learning of actions in video using video scripts or event order has been addressed in [3, 4, 5, 9, 16].”
Extreme Clicking for Efficient Object Annotation (2017, background): “Alternative types of supervision information such as eye-tracking data [44, 46], text from news articles or web pages [17, 28], or even movie scripts [7] have also been explored.”
MovieQA: Understanding Stories in Movies through Question-Answering (2015, background): “For TV series in particular, the majority of work aims at recognizing and tracking characters in the videos [2, 4, 28, 35].”
Unsupervised Learning by Predicting Noise (2017, methodology): “It has been successfully applied to several computer vision applications, like object discovery (Joulin et al., 2010; Tang et al., 2014) or video/text alignment (Bojanowski et al., 2013; 2014; Ra-manathan et al., 2014).”
A dataset for Movie Description (2015, background): “[10] attacks the problem of learning a joint model of actors and actions in movies using weak supervision provided by scripts.”
Weakly Supervised Action Labeling in Videos under Ordering Constraints (2014, methodology): “Nonetheless, we see that making use of ordering constraints as used by our method signigicantly improves over simple linear inequalities (“at least one” constraints as formulated in [5]).”
Weakly-Supervised Alignment of Video with Text (2015, methodology): “Contrary to standard clustering techniques, it uses a discriminative cost function [2, 16] and it has been used in image co-segmentation [20, 21], object colocalization [22], person identification in video [6, 36], and alignment of labels to videos [7].”
Connectionist Temporal Modeling for Weakly Supervised Action Labeling (2016, background): “An alternative is to develop methods that require weak supervision, which may be automatically extracted from movie scripts [3,5,9] or instructional videos [2,4,50] at a lower cost.”
Large-Scale Weakly-Supervised Pre-Training for Video Action Recognition (2019, background): “[8, 21, 23, 47] analyzed movie scripts for automatic annotation of human actions for recognizing and localizing actions and actors.”
Domain-Adaptive Discriminative One-Shot Learning of Gestures (2014, background): “TV broadcasts with aligned subtitles (or similarly actions from movies with aligned transcripts [1, 3, 14]).”
Instance-Level Video Segmentation from Object Tracks (2016, methodology): “Discriminative clustering has recently been applied to several problems including image co-segmentation [24, 25], object co-localization in images [42], finding actor identities in movies [6, 35] and temporal action localization [7].”
Cross-Modal Supervision for Learning Active Speaker Detection in Video (2016, unknown): “The work of Bojanowski et al. [13] is another example of one mode of information weakly supervising another.”
Y Qualification Main task Y Return N N Y User qualified ? N Feedback Pass ? Pass ? Annotator training Annotating images (2017, background): “Alternative types of supervision information such as eye-tracking data [44, 46], text from news articles or web pages [17, 28], or even movie scripts [7] have also been explored.”
It ’ s in the bag : Stronger supervision for automated face labelling (2015, methodology): “This topic has received plenty of attention in the community and papers have investigated the problem under various assumptions on the available information [2, 3, 5, 4, 6, 8, 9, 12, 17, 18, 20, 22, 24].”
Learning from Video and Text via Large-Scale Discriminative Clustering (2017, methodology): “While actors are learned separately for each movie, differently from [4], our method simultaneously learns actions from all movies and movie scripts available for training.”
A flexible model for training action localization with varying levels of supervision (2018, methodology): “In this paper, we employ the method proposed in [3] which is appealing due to its simple analytic form, its flexibility and its recent successes for weakly supervised methods in computer vision [1, 4, 17].”
Who's that Actor? Automatic Labelling of Actors in TV Series Starting from IMDB Images (2016, methodology): “Finally, action recognition has been used as an additional cue as well, along with weak supervision from transcripts [11].”