Learning from Video and Text via Large-Scale Discriminative Clustering

By Antoine Miech, Jean-Baptiste Alayrac, Piotr Bojanowski, Ivan Laptev, Josef Sivic. 2017

45 citations recorded by the source corpus.

Discriminative clustering has been successfully applied to a number of weakly supervised learning tasks. Such applications include person and action recognition, text-to-video alignment, object co-segmentation and co-localization in videos and images. One drawback of discriminative clustering, however, is its limited scalability. We address this issue and propose an online optimization algorithm based on the Block-Coordinate Frank-Wolfe algorithm. We apply the proposed method to the problem of weakly supervised learning of actions and actors from movies together with corresponding movie scripts. The scaling up of the learning problem to 66 feature-length movies enables us to significantly improve weakly supervised action recognition.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, neutral): “Multiple instance learning methods have been employed in many weakly-supervised video understanding problems including: person recognition in movies using scripts [12, 50, 61], anomaly detection [70], weakly supervised action classification [47, 68] and localization [17, 22, 82], co-reference resolu”
A Joint Sequence Fusion Model for Video Question Answering and Retrieval (2018, neutral): “ VSE-LSTM [19] 3.1 3.8 10.4 12.7 16.5 17.1 79 66 EITanque [30] 4.7 4.7 15.9 16.6 23.4 24.1 64 41 SNUVL [29] 3.6 3.5 14.7 15.9 23.9 23.8 50 44 CT-SAN [9] 4.5 4.4 14.1 16.6 20.9 22.3 67 35 Miech et al. [44] 7.3 – 19.2 – 27.1 – 52 – JSTfc 4.7 5.1 17.2 21.1 25.2 29.1 52 30 JSTlstm 7.6 9.2 19.2 28.2 27.1”
Learning a Text-Video Embedding from Incomplete and Heterogeneous Data (2018, background): “Towards this goal recent works study representations of vision and language addressing tasks such as visual question answering [1], [2], action learning and discovery [3], [4], [5], text-based event localization [6] as well as video captioning, retrieval and summarization [7], [8], [9], [10].”
Weakly-Supervised Learning of Visual Relations (2017, methodology): “We optimize the cost in (3) on pairs of objects in the training set using a variant of the Frank-Wolfe al-gorithm [33, 36].”
A flexible model for training action localization with varying levels of supervision (2018, methodology): “Recently, [28] proposed a block coordinate [21] version of Frank-Wolfe for discriminative clustering.”
Temporal Localization of Moments in Video Collections with Natural Language (2019, methodology): “Datasets of videos with temporally aligned text [10, 16, 17, 22, 29] have been used for aligning movie scripts, textual instructions, and sentences in a paragraph with a single video [3, 25, 32, 39], video object segmentation [20], and retrieving moments in a single video given a text query [4, 5, 1”
Projection-Free Optimization on Uniformly Convex Sets (2020, methodology): “…2010; Jaggi, 2011) used for instance in large-scale structured prediction (Bojanowski et al., 2014, 2015; Alayrac et al., 2016; Seguin et al., 2016; Miech et al., 2017; Peyre et al., 2017; Miech et al., 2018), quadrature rules in RKHS (Bach et al., 2012; Lacoste-Julien et al., 2015; Futami et al.,…”
Speech2Action: Cross-Modal Supervision for Action Recognition (2020, background): “Hence previous works have been limited to a small scale, no more than tens of movies or a season of a TV series [2, 8, 22, 25, 26].”
Learning from Narrated Instruction Videos (2015, methodology): “This is possible as our constraints, similar to [25], decompose over the different videos.”